2024-03-29T11:21:27.333 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:21:27.334 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/orig.config.yaml 2024-03-29T11:21:27.371 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:21:27.456 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622724' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTYl8mD0g8ZLBN36uwr+/8pyQ1eLH6j5qG+Sx0y7nd16YU3Vam4ldskPPXd7k656VJQ76Tk/gP4GQaHQDxxtpI= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCyO5V/BD/RsJnRH35tRBrBritxo2PqbBiQAiauzZgHMv0s0pwjuu5tEwE6RvCj+d/pKhHmHH9wYy3Nqxd1u0U0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:21:27.456 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:21:27.457 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:21:27.457 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:21:27.458 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:21:27.483 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:21:27.483 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:21:27.483 INFO:teuthology.packaging:ref: None 2024-03-29T11:21:27.484 INFO:teuthology.packaging:tag: None 2024-03-29T11:21:27.484 INFO:teuthology.packaging:branch: main 2024-03-29T11:21:27.484 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:21:27.484 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:21:27.774 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:21:27.776 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:21:27.784 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:21:27.785 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:21:27.797 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:21:27.811 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:21:27.821 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:21:27.844 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:42.275417', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTYl8mD0g8ZLBN36uwr+/8pyQ1eLH6j5qG+Sx0y7nd16YU3Vam4ldskPPXd7k656VJQ76Tk/gP4GQaHQDxxtpI='} 2024-03-29T11:21:27.864 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:42.276703', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCyO5V/BD/RsJnRH35tRBrBritxo2PqbBiQAiauzZgHMv0s0pwjuu5tEwE6RvCj+d/pKhHmHH9wYy3Nqxd1u0U0='} 2024-03-29T11:21:27.864 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:21:27.871 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:21:27.872 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:21:27.872 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:21:27.949 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc9d08a70d0>, signals=[15]) 2024-03-29T11:21:27.949 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:21:27.957 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:21:27.958 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:21:27.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:28.071 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:21:28.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:28.163 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:21:28.174 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-03-29T11:21:28.193 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-03-29T11:21:28.193 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:28.251 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:28.251 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-03-29T11:21:28.274 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-03-29T11:21:28.291 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-03-29T11:21:28.291 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-03-29T11:21:28.347 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-03-29T11:21:28.347 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-03-29T11:21:28.347 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-03-29T11:21:28.347 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:28.347 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:28.348 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:28.348 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-03-29T11:21:28.381 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:21:28.396 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:21:28.416 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:21:28.417 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:28.420 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:28.435 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:21:28.448 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:21:28.450 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:28.477 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:28.595 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:21:28.646 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:21:28.669 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:21:28.669 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:21:28.670 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:28.670 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:28.687 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-29T11:21:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:28.688 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-29T11:21:28.744 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:28.744 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-361.el9.x86_64 2024-03-29T11:21:28.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-03-29T11:21:28.746 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:28.746 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-361.el9.x86_64 2024-03-29T11:21:28.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-03-29T11:21:30.879 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:21:30.932 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:21:32.305 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-29T11:21:32.658 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-29T11:21:33.487 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-29T11:21:35.230 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-29T11:21:39.499 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-29T11:21:40.922 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-03-29T11:21:41.989 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 917 kB/s | 8.0 MB 00:08 2024-03-29T11:21:42.693 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:21:44.590 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T11:21:47.798 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-29T11:21:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:21:50.155 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:21:50.252 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:21:50.262 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-03-29T11:21:50.263 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:21:50.264 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 76 M 2024-03-29T11:21:50.264 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 123 M 2024-03-29T11:21:50.264 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:21:50.425 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:21:50.992 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:21:51.567 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T11:21:52.050 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:21:52.053 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:52.053 INFO:teuthology.orchestra.run.smithi045.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:21:52.053 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:21:52.504 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:21:52.504 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:21:53.385 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:21:53.385 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:21:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-29T11:21:55.307 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-29T11:21:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:21:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:21:56.526 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:21:57.050 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:21:59.411 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:21:59.543 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:02.083 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-29T11:22:04.407 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:04.407 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:04.506 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:22:04.514 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-03-29T11:22:04.515 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:04.516 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 76 M 2024-03-29T11:22:04.516 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 123 M 2024-03-29T11:22:04.516 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:22:04.649 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:04.668 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:22:04.691 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:05.251 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:22:05.802 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T11:22:06.352 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T11:22:06.355 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:06.355 INFO:teuthology.orchestra.run.smithi190.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-29T11:22:06.355 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:22:06.721 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:22:06.721 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:22:07.612 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:22:07.612 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:22:09.822 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:09.916 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:22:10.593 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:10.693 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:13.441 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:13.580 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:18.258 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:18.304 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:41.003 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:41.003 INFO:teuthology.orchestra.run.smithi045.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:41.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:22:41.048 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:41.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:22:42.145 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:42.252 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.253 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:42.253 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:42.253 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:42.253 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:42.254 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:22:42.254 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:42.254 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:42.337 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:42.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:42.385 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:22:42.386 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:42.387 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:42.387 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:42.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-03-29T11:22:44.498 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:22:44.536 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-03-29T11:22:44.538 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T11:22:44.593 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.MTqGvXKsWB 2024-03-29T11:22:44.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MTqGvXKsWB 2024-03-29T11:22:44.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.MTqGvXKsWB 2024-03-29T11:22:44.802 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.MTqGvXKsWB is 6KB 2024-03-29T11:22:44.851 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.MTqGvXKsWB 2024-03-29T11:22:44.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:22:44.934 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:22:44.934 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:22:44.934 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:22:44.934 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:22:44.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:22:45.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-03-29T11:22:52.445 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.445 INFO:teuthology.orchestra.run.smithi190.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:52.445 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:52.489 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.857 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.857 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:52.857 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:52.857 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:53.238 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:22:53.575 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:53.676 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.676 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.676 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.676 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:53.677 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:53.677 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:22:53.677 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:53.677 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:53.756 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.757 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-03-29T11:22:53.800 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:22:53.802 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:53.803 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:53.803 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:54.371 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-03-29T11:22:55.792 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:22:55.827 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-03-29T11:22:55.829 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-29T11:22:55.844 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.kWzVWNubMa 2024-03-29T11:22:55.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kWzVWNubMa 2024-03-29T11:22:55.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.kWzVWNubMa 2024-03-29T11:22:56.056 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.kWzVWNubMa is 6KB 2024-03-29T11:22:56.104 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.kWzVWNubMa 2024-03-29T11:22:56.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:22:56.183 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:22:56.183 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:22:56.184 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:22:56.184 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:22:56.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:22:56.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-03-29T11:23:15.101 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:23:15.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:23:15.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:26.349 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:23:26.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-29T11:23:26.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:33.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-03-29T11:23:42.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:23:42.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:45.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-03-29T11:23:57.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:23:57.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:26.379 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:24:29.358 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-29T11:24:29.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:24:29.899 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:24:29.899 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-29T11:24:29.913 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:24:29.913 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:24:29.913 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:24:29.914 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:24:30.915 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:24:30.915 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:24:31.012 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-03-29T11:24:31.045 DEBUG:teuthology.parallel:result is None 2024-03-29T11:24:35.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-29T11:24:35.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:35.826 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-29T11:24:36.225 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-29T11:24:36.225 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:24:36.225 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-29T11:24:36.239 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:24:36.240 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:24:36.240 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:24:36.240 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:24:37.241 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:24:37.241 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:24:37.299 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-03-29T11:24:37.335 DEBUG:teuthology.parallel:result is None 2024-03-29T11:24:37.335 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:24:37.347 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:24:37.347 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:24:37.350 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:24:37.365 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:24:37.377 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:24:37.390 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:24:37.391 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:37.408 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:37.439 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:24:37.447 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:24:37.448 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:24:37.475 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:24:37.505 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:37.507 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:37.519 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:37.520 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:37.522 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:24:37.532 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:24:37.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:24:37.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:24:37.591 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:24:37.601 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:24:37.601 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:24:37.634 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:24:37.648 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:37.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:37.782 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:37.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:37.907 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:24:37.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:24:37.974 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:38.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:38.065 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:38.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:38.190 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:24:38.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:24:38.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-03-29T11:24:38.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-03-29T11:24:38.321 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:38.353 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:38.636 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:24:38.646 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:24:38.647 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:24:38.670 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:24:38.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-03-29T11:24:38.761 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-03-29T11:24:38.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-03-29T11:24:38.812 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-03-29T11:24:38.813 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:24:38.814 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-03-29T11:24:38.839 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-03-29T11:24:38.840 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-03-29T11:24:38.856 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-03-29T11:24:38.856 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:24:38.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:38.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:38.936 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:24:38.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:24:38.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:24:39.016 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:24:39.028 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:24:39.045 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:24:39.046 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:28:58.002 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-03-29T11:28:58.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:28:58.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:58.099 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-29T11:28:58.173 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-29T11:28:58.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-29T11:28:58.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:58.263 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-29T11:28:58.348 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-29T11:28:58.349 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:28:58.360 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:28:58.360 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:28:58.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:28:58.363 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:28:58.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:28:58.394 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:28:58.412 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:28:58.418 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:28:58.435 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:28:58.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-03-29T11:28:58.457 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-03-29T11:28:58.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-03-29T11:28:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:28:58.478 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-03-29T11:28:58.490 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:28:58.495 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:28:58.511 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:28:58.541 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:28:58.544 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:28:58.544 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-29T11:28:58.544 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.545 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.545 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.545 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.563 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.567 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:58.568 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:28:58.577 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:28:58.577 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:28:58.578 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:28:58.578 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:28:58.583 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:28:58.583 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:28:58.584 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:28:58.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:28:58.587 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:28:58.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:28:58.826 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:28:58.826 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:28:58.829 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:28:58.830 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:28:58.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:28:58.936 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:28:58.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:28:58.943 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:28:58.943 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:28:58.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:28:58.966 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:28:58.966 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:28:58.971 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:28:58.971 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:28:59.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:28:59.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:28:59.115 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:28:59.117 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:28:59.181 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-03-29T11:28:59.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-03-29T11:28:59.183 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-03-29T11:28:59.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-03-29T11:28:59.432 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-03-29T11:28:59.434 INFO:teuthology.orchestra.run.smithi190.stdout:74 files removed 2024-03-29T11:28:59.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-03-29T11:28:59.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-03-29T11:29:00.205 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 186 kB/s | 82 kB 00:00 2024-03-29T11:29:00.229 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 177 kB/s | 82 kB 00:00 2024-03-29T11:29:00.630 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-29T11:29:00.655 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T11:29:00.997 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-29T11:29:01.081 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-29T11:29:01.691 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T11:29:02.133 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-03-29T11:29:03.117 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-29T11:29:03.642 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:29:04.335 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.0 MB 00:00 2024-03-29T11:29:04.391 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-29T11:29:06.425 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-29T11:29:10.985 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-29T11:29:12.778 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-29T11:29:14.395 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:29:19.579 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T11:29:22.289 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:29:22.292 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:22.292 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:29:22.293 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:22.293 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:29:22.293 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:29:22.293 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-29T11:29:22.293 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:29:22.294 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:29:22.294 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:29:22.294 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options 2024-03-29T11:29:22.294 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:29:22.294 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:29:22.295 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:29:22.296 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:29:22.297 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:29:22.298 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:29:22.298 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:29:22.298 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:29:22.298 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-03-29T11:29:22.302 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout:Install 30 Packages 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-03-29T11:29:22.303 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:22.304 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2024-03-29T11:29:22.304 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:29:23.446 INFO:teuthology.orchestra.run.smithi045.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.4 MB/s | 5.3 MB 00:00 2024-03-29T11:29:23.505 INFO:teuthology.orchestra.run.smithi045.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 433 kB/s | 25 kB 00:00 2024-03-29T11:29:23.614 INFO:teuthology.orchestra.run.smithi045.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-29T11:29:23.714 INFO:teuthology.orchestra.run.smithi045.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T11:29:23.997 INFO:teuthology.orchestra.run.smithi045.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.3 MB/s | 9.7 MB 00:01 2024-03-29T11:29:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-29T11:29:24.181 INFO:teuthology.orchestra.run.smithi045.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:29:24.265 INFO:teuthology.orchestra.run.smithi045.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.3 MB/s | 5.1 MB 00:00 2024-03-29T11:29:24.290 INFO:teuthology.orchestra.run.smithi045.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.5 MB/s | 162 kB 00:00 2024-03-29T11:29:24.357 INFO:teuthology.orchestra.run.smithi045.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.4 MB/s | 320 kB 00:00 2024-03-29T11:29:24.382 INFO:teuthology.orchestra.run.smithi045.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.2 MB/s | 298 kB 00:00 2024-03-29T11:29:24.441 INFO:teuthology.orchestra.run.smithi045.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:29:24.508 INFO:teuthology.orchestra.run.smithi045.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 842 kB/s | 106 kB 00:00 2024-03-29T11:29:24.533 INFO:teuthology.orchestra.run.smithi045.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-29T11:29:24.588 INFO:teuthology.orchestra.run.smithi045.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-29T11:29:24.742 INFO:teuthology.orchestra.run.smithi045.stdout:(16/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.7 MB/s | 20 MB 00:02 2024-03-29T11:29:24.768 INFO:teuthology.orchestra.run.smithi045.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 661 kB 00:00 2024-03-29T11:29:24.793 INFO:teuthology.orchestra.run.smithi045.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 282 kB 00:00 2024-03-29T11:29:24.843 INFO:teuthology.orchestra.run.smithi045.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 415 kB/s | 41 kB 00:00 2024-03-29T11:29:24.868 INFO:teuthology.orchestra.run.smithi045.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 465 kB/s | 35 kB 00:00 2024-03-29T11:29:24.894 INFO:teuthology.orchestra.run.smithi045.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-03-29T11:29:24.919 INFO:teuthology.orchestra.run.smithi045.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 616 kB/s | 46 kB 00:00 2024-03-29T11:29:25.044 INFO:teuthology.orchestra.run.smithi045.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-29T11:29:25.144 INFO:teuthology.orchestra.run.smithi045.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-29T11:29:25.170 INFO:teuthology.orchestra.run.smithi045.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-29T11:29:25.220 INFO:teuthology.orchestra.run.smithi045.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-29T11:29:25.354 INFO:teuthology.orchestra.run.smithi045.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T11:29:25.379 INFO:teuthology.orchestra.run.smithi045.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:29:25.421 INFO:teuthology.orchestra.run.smithi045.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-29T11:29:25.462 INFO:teuthology.orchestra.run.smithi045.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T11:29:25.763 INFO:teuthology.orchestra.run.smithi045.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.8 MB/s | 3.4 MB 00:00 2024-03-29T11:29:25.813 INFO:teuthology.orchestra.run.smithi045.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.0 MB/s | 3.1 MB 00:00 2024-03-29T11:29:25.817 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:25.817 INFO:teuthology.orchestra.run.smithi045.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-29T11:29:25.817 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:29:25.860 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:29:25.860 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:29:26.362 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:29:26.362 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:29:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 863 kB/s | 19 MB 00:22 2024-03-29T11:29:28.007 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:29:28.074 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:29:28.101 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:29:28.126 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:29:28.149 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:28.166 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:28.392 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:28.448 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:28.844 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:29:28.878 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:29.087 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:29.177 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:29:29.213 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:29.259 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:29.288 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:29:29.332 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:29:29.401 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:29.487 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:29.522 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:29:29.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:29:29.666 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:29:29.706 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:29:29.738 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:29:29.767 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:29:29.792 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:29:30.118 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:29:30.146 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:29:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:29:30.786 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:29:30.815 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:30.887 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:31.075 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:29:31.109 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:29:31.907 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T11:29:33.392 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:33.566 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T11:29:34.013 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:34.060 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:34.135 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:34.135 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:29:34.135 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:29:34.136 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:34.943 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:29:39.909 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-29T11:29:42.485 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:29:42.500 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:29:42.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:29:42.502 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:29:42.503 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout:Install 30 Packages 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-03-29T11:29:42.504 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:29:42.505 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 58 M 2024-03-29T11:29:42.505 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:29:43.277 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.1 MB/s | 5.3 MB 00:00 2024-03-29T11:29:43.336 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 433 kB/s | 25 kB 00:00 2024-03-29T11:29:43.419 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-29T11:29:43.494 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T11:29:43.870 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-29T11:29:43.929 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 773 kB/s | 45 kB 00:00 2024-03-29T11:29:43.995 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:29:44.071 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T11:29:44.154 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T11:29:44.388 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 12 MB/s | 20 MB 00:01 2024-03-29T11:29:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:44.389 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-03-29T11:29:44.389 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-03-29T11:29:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:44.413 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 298 kB 00:00 2024-03-29T11:29:44.438 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T11:29:44.555 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-29T11:29:44.581 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T11:29:44.597 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:29:44.648 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:29:44.673 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:29:44.690 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 994 kB/s | 41 kB 00:00 2024-03-29T11:29:44.715 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T11:29:44.732 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 840 kB/s | 35 kB 00:00 2024-03-29T11:29:44.758 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T11:29:45.141 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 757 kB/s | 309 kB 00:00 2024-03-29T11:29:45.216 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T11:29:45.283 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-03-29T11:29:45.317 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:45.342 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:45.342 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:45.342 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:29:45.342 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:45.342 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:45.343 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:45.344 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:45.367 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-03-29T11:29:45.429 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:45.429 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:45.433 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-03-29T11:29:45.542 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 9.7 MB 00:02 2024-03-29T11:29:45.567 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-29T11:29:45.618 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-29T11:29:45.843 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-29T11:29:45.968 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 9.6 MB/s | 3.1 MB 00:00 2024-03-29T11:29:46.143 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-03-29T11:29:46.151 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:46.151 INFO:teuthology.orchestra.run.smithi190.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-29T11:29:46.151 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:29:46.189 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:29:46.189 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:29:46.497 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:46.498 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:29:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:29:46.500 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:29:46.661 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:29:46.662 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:29:47.514 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:47.514 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:47.514 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-29T11:29:47.514 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:29:47.515 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:29:47.516 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:29:47.517 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:29:47.517 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:29:47.517 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:29:47.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:47.517 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:29:47.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-03-29T11:29:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:29:48.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:29:48.223 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:29:48.249 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:29:48.271 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:48.288 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:48.409 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:29:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:48.576 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:29:48.576 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:48.576 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:29:48.576 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-29T11:29:48.577 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:29:48.578 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 48 M 2024-03-29T11:29:48.578 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 204 M 2024-03-29T11:29:48.578 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:29:48.815 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:29:48.866 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:49.032 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:29:49.077 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:49.177 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:29:49.195 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 468 kB/s | 217 kB 00:00 2024-03-29T11:29:49.212 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:29:49.281 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:49.304 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:49.367 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:49.396 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-29T11:29:49.402 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:29:49.421 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 237 kB/s | 163 kB 00:00 2024-03-29T11:29:49.446 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:29:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:29:49.553 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:29:49.563 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:29:49.594 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:29:49.635 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:29:49.664 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:29:49.696 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:29:50.024 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:29:50.051 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:50.125 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:50.156 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:29:50.201 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:29:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:29:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:50.758 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:29:50.864 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:29:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:53.619 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:53.655 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:53.748 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:53.748 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:29:53.748 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:29:53.748 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:29:53.790 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-03-29T11:30:03.534 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-03-29T11:30:03.535 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-03-29T11:30:03.535 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:04.447 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:30:04.472 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:04.473 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:30:04.549 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:30:04.549 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:05.004 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 47 MB 00:16 2024-03-29T11:30:05.007 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:05.008 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.9 MB/s | 48 MB 00:16 2024-03-29T11:30:05.022 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:05.045 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:05.045 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:05.448 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:05.686 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:30:05.687 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:30:05.688 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:30:05.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:30:06.036 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:06.106 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:30:06.162 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:30:06.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:30:06.247 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:30:06.274 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:30:06.734 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.735 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:30:06.736 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:30:06.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:06.737 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:07.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-03-29T11:30:07.675 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:07.828 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:30:07.829 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-29T11:30:07.830 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:07.831 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 48 M 2024-03-29T11:30:07.832 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 204 M 2024-03-29T11:30:07.832 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:08.111 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:30:08.161 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:30:08.195 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:30:08.320 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 443 kB/s | 163 kB 00:00 2024-03-29T11:30:08.420 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-29T11:30:10.720 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-29T11:30:10.724 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:10.724 INFO:teuthology.orchestra.run.smithi190.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T11:30:10.740 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:10.761 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:10.761 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:11.005 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:30:11.085 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:30:11.137 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:11.137 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:11.721 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:11.795 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:30:11.840 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:30:11.871 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:30:11.921 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:30:11.949 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:30:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:30:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:30:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:30:13.536 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:30:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:30:13.537 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:30:13.904 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:13.905 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:30:13.905 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:30:13.905 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:30:13.905 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:14.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-03-29T11:30:14.772 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:14.935 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T11:30:14.937 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:30:14.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:30:14.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:14.940 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-29T11:30:14.941 INFO:teuthology.orchestra.run.smithi045.stdout:Install 36 Packages 2024-03-29T11:30:14.941 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:14.941 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 30 M 2024-03-29T11:30:14.941 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 107 M 2024-03-29T11:30:14.941 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:15.759 INFO:teuthology.orchestra.run.smithi045.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-03-29T11:30:15.993 INFO:teuthology.orchestra.run.smithi045.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-03-29T11:30:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:30:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:30:16.535 INFO:teuthology.orchestra.run.smithi045.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2024-03-29T11:30:16.835 INFO:teuthology.orchestra.run.smithi045.stdout:(4/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-29T11:30:16.860 INFO:teuthology.orchestra.run.smithi045.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 761 kB/s | 247 kB 00:00 2024-03-29T11:30:17.080 INFO:teuthology.orchestra.run.smithi045.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 764 kB/s | 187 kB 00:00 2024-03-29T11:30:17.111 INFO:teuthology.orchestra.run.smithi045.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 577 kB/s | 144 kB 00:00 2024-03-29T11:30:17.136 INFO:teuthology.orchestra.run.smithi045.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 565 kB/s | 32 kB 00:00 2024-03-29T11:30:17.220 INFO:teuthology.orchestra.run.smithi045.stdout:(9/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 4.7 MB 00:01 2024-03-29T11:30:17.246 INFO:teuthology.orchestra.run.smithi045.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 352 kB/s | 47 kB 00:00 2024-03-29T11:30:17.271 INFO:teuthology.orchestra.run.smithi045.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 288 kB/s | 39 kB 00:00 2024-03-29T11:30:17.313 INFO:teuthology.orchestra.run.smithi045.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-29T11:30:17.455 INFO:teuthology.orchestra.run.smithi045.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 5.5 MB/s | 1.1 MB 00:00 2024-03-29T11:30:17.480 INFO:teuthology.orchestra.run.smithi045.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 493 kB/s | 103 kB 00:00 2024-03-29T11:30:17.513 INFO:teuthology.orchestra.run.smithi045.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 620 kB/s | 124 kB 00:00 2024-03-29T11:30:17.539 INFO:teuthology.orchestra.run.smithi045.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-03-29T11:30:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 830 kB/s | 125 kB 00:00 2024-03-29T11:30:17.739 INFO:teuthology.orchestra.run.smithi045.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-29T11:30:17.765 INFO:teuthology.orchestra.run.smithi045.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 328 kB/s | 43 kB 00:00 2024-03-29T11:30:17.790 INFO:teuthology.orchestra.run.smithi045.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-29T11:30:17.832 INFO:teuthology.orchestra.run.smithi045.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T11:30:17.857 INFO:teuthology.orchestra.run.smithi045.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-29T11:30:17.908 INFO:teuthology.orchestra.run.smithi045.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-29T11:30:17.933 INFO:teuthology.orchestra.run.smithi045.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 483 kB/s | 23 kB 00:00 2024-03-29T11:30:17.950 INFO:teuthology.orchestra.run.smithi045.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 464 kB/s | 19 kB 00:00 2024-03-29T11:30:17.975 INFO:teuthology.orchestra.run.smithi045.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 470 kB/s | 20 kB 00:00 2024-03-29T11:30:17.992 INFO:teuthology.orchestra.run.smithi045.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T11:30:18.026 INFO:teuthology.orchestra.run.smithi045.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.6 MB/s | 79 kB 00:00 2024-03-29T11:30:18.059 INFO:teuthology.orchestra.run.smithi045.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:30:18.085 INFO:teuthology.orchestra.run.smithi045.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 283 kB/s | 16 kB 00:00 2024-03-29T11:30:18.110 INFO:teuthology.orchestra.run.smithi045.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T11:30:18.135 INFO:teuthology.orchestra.run.smithi045.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-29T11:30:18.160 INFO:teuthology.orchestra.run.smithi045.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:30:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-29T11:30:18.244 INFO:teuthology.orchestra.run.smithi045.stdout:(35/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-03-29T11:30:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T11:30:18.276 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:18.276 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.9 MB/s | 30 MB 00:03 2024-03-29T11:30:18.301 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:18.347 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:18.348 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:18.755 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:18.755 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:18.787 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:30:18.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:30:18.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:30:18.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:30:18.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:30:18.788 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:30:19.148 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:30:19.148 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:19.148 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:19.148 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:19.149 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:19.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-03-29T11:30:20.029 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:20.123 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:20.189 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:30:20.200 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:20.202 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-29T11:30:20.202 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:20.202 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-29T11:30:20.202 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:30:20.203 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:30:20.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:20.205 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-29T11:30:20.206 INFO:teuthology.orchestra.run.smithi190.stdout:Install 36 Packages 2024-03-29T11:30:20.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:20.206 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-03-29T11:30:20.206 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 107 M 2024-03-29T11:30:20.207 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:20.223 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:30:20.250 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:30:20.324 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:30:20.374 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:30:20.407 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:30:20.506 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:30:20.588 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:30:20.636 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:30:20.710 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:30:20.770 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:30:20.816 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:30:20.851 INFO:teuthology.orchestra.run.smithi190.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-29T11:30:20.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:30:20.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:30:21.126 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:30:21.170 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:30:21.193 INFO:teuthology.orchestra.run.smithi190.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-29T11:30:21.209 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:30:21.226 INFO:teuthology.orchestra.run.smithi190.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 1.5 MB 00:00 2024-03-29T11:30:21.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:30:21.277 INFO:teuthology.orchestra.run.smithi190.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.8 MB/s | 247 kB 00:00 2024-03-29T11:30:21.290 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:30:21.370 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:30:21.410 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:21.443 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:30:21.483 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:30:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:30:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:30:21.569 INFO:teuthology.orchestra.run.smithi190.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 641 kB/s | 187 kB 00:00 2024-03-29T11:30:21.619 INFO:teuthology.orchestra.run.smithi190.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:30:21.646 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:30:21.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:30:21.686 INFO:teuthology.orchestra.run.smithi190.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 472 kB/s | 32 kB 00:00 2024-03-29T11:30:21.736 INFO:teuthology.orchestra.run.smithi190.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-29T11:30:21.760 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:30:21.778 INFO:teuthology.orchestra.run.smithi190.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-03-29T11:30:21.828 INFO:teuthology.orchestra.run.smithi190.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:30:21.902 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:30:21.945 INFO:teuthology.orchestra.run.smithi190.stdout:(11/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.3 MB/s | 4.7 MB 00:01 2024-03-29T11:30:21.979 INFO:teuthology.orchestra.run.smithi190.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-03-29T11:30:22.004 INFO:teuthology.orchestra.run.smithi190.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-29T11:30:22.029 INFO:teuthology.orchestra.run.smithi190.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:30:22.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:22.055 INFO:teuthology.orchestra.run.smithi190.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:30:22.122 INFO:teuthology.orchestra.run.smithi190.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-29T11:30:22.220 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:30:22.247 INFO:teuthology.orchestra.run.smithi190.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-03-29T11:30:22.247 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:22.274 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:22.406 INFO:teuthology.orchestra.run.smithi190.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 532 kB/s | 151 kB 00:00 2024-03-29T11:30:22.431 INFO:teuthology.orchestra.run.smithi190.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 236 kB/s | 43 kB 00:00 2024-03-29T11:30:22.498 INFO:teuthology.orchestra.run.smithi190.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T11:30:22.549 INFO:teuthology.orchestra.run.smithi190.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:30:22.599 INFO:teuthology.orchestra.run.smithi190.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-29T11:30:22.616 INFO:teuthology.orchestra.run.smithi190.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-03-29T11:30:22.649 INFO:teuthology.orchestra.run.smithi190.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T11:30:22.675 INFO:teuthology.orchestra.run.smithi190.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-03-29T11:30:22.708 INFO:teuthology.orchestra.run.smithi190.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-03-29T11:30:22.759 INFO:teuthology.orchestra.run.smithi190.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-03-29T11:30:22.784 INFO:teuthology.orchestra.run.smithi190.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T11:30:22.943 INFO:teuthology.orchestra.run.smithi190.stdout:(29/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 9.6 MB/s | 17 MB 00:01 2024-03-29T11:30:22.968 INFO:teuthology.orchestra.run.smithi190.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 89 kB/s | 16 kB 00:00 2024-03-29T11:30:22.993 INFO:teuthology.orchestra.run.smithi190.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-03-29T11:30:23.027 INFO:teuthology.orchestra.run.smithi190.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-29T11:30:23.069 INFO:teuthology.orchestra.run.smithi190.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T11:30:23.110 INFO:teuthology.orchestra.run.smithi190.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T11:30:23.136 INFO:teuthology.orchestra.run.smithi190.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-29T11:30:23.153 INFO:teuthology.orchestra.run.smithi190.stdout:(36/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 429 kB/s | 90 kB 00:00 2024-03-29T11:30:23.156 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:23.156 INFO:teuthology.orchestra.run.smithi190.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T11:30:23.180 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:23.220 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:23.220 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:23.646 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:23.646 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:23.825 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:23.825 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:23.826 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:30:23.826 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:23.826 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:23.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:24.542 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:24.565 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:24.566 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:24.566 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:30:24.566 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:24.566 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:24.566 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:24.634 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:25.047 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:30:25.081 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:30:25.116 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:30:25.186 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:30:25.240 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:30:25.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:30:25.364 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:30:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:30:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:30:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:30:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:30:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:30:25.697 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:30:25.752 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:30:25.944 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:30:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:30:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:30:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:30:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:30:26.183 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:30:26.216 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:26.241 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:30:26.281 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:30:26.316 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:30:26.351 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:30:26.408 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:26.408 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:30:26.409 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:30:26.410 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:30:26.411 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:30:26.412 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:30:26.441 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:30:26.474 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:30:26.546 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:30:26.684 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:30:26.809 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:26.965 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:30:26.995 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:27.021 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:27.249 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:30:27.250 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:30:27.251 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:27.252 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:27.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-03-29T11:30:28.189 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:28.291 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:28.346 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:28.347 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:30:28.347 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:28.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-03-29T11:30:28.446 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:28.473 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:28.473 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:28.473 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:30:28.473 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:28.473 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:28.474 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:28.906 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:29.012 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:29.163 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:29.163 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:29.164 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:30:29.164 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:29.164 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:29.164 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:29.169 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:29.170 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:29.170 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 747 k 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 754 k 2024-03-29T11:30:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:29.221 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:29.573 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.8 MB/s | 747 kB 00:00 2024-03-29T11:30:29.574 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:29.574 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.8 MB/s | 747 kB 00:00 2024-03-29T11:30:29.574 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:29.580 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:29.580 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:29.586 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:29.586 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:29.665 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:30.058 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:30.156 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:30.516 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:30.827 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:31.000 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:31.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:30:31.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:30:31.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:30:31.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:30:31.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:30:31.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:30:31.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:31.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:30:31.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:30:31.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:30:31.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:30:31.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:30:31.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:30:31.585 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:31.742 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:31.743 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 145 k 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 436 k 2024-03-29T11:30:31.744 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:30:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:30:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:30:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:30:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:31.939 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:32.131 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 376 kB/s | 145 kB 00:00 2024-03-29T11:30:32.131 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:32.132 INFO:teuthology.orchestra.run.smithi045.stdout:Total 375 kB/s | 145 kB 00:00 2024-03-29T11:30:32.132 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:32.139 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:32.139 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:32.194 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:32.194 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:32.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-03-29T11:30:32.348 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:32.398 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:32.422 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:32.422 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:30:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:32.826 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:32.924 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:32.977 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:32.978 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:30:32.978 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:33.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-03-29T11:30:33.369 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:33.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:33.369 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:33.369 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:33.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:33.370 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:33.602 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:33.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-03-29T11:30:33.753 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:30:33.754 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:33.755 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 747 k 2024-03-29T11:30:33.755 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 754 k 2024-03-29T11:30:33.755 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:34.181 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:34.283 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:34.292 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.4 MB/s | 747 kB 00:00 2024-03-29T11:30:34.293 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:34.293 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.4 MB/s | 747 kB 00:00 2024-03-29T11:30:34.293 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:34.298 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:34.298 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:34.304 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:34.304 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:34.338 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:34.339 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:30:34.339 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:34.380 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:34.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:30:34.825 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:34.988 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:35.144 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:35.145 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:30:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-29T11:30:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:35.148 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.9 M 2024-03-29T11:30:35.148 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-29T11:30:35.148 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:35.287 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:35.560 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 127 kB/s | 31 kB 00:00 2024-03-29T11:30:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:35.572 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:35.573 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:35.585 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-29T11:30:35.727 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:30:35.761 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:30:35.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:30:35.920 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-29T11:30:36.053 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:30:36.070 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 129 kB/s | 19 kB 00:00 2024-03-29T11:30:36.187 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-03-29T11:30:36.204 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-29T11:30:36.252 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 738 kB/s | 31 kB 00:00 2024-03-29T11:30:36.295 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:30:36.347 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:36.480 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.5 MB/s | 1.8 MB 00:01 2024-03-29T11:30:36.485 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:36.485 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-03-29T11:30:36.499 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:30:36.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:36.502 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:36.502 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:30:36.502 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:30:36.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:36.503 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 145 k 2024-03-29T11:30:36.503 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 436 k 2024-03-29T11:30:36.503 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:36.504 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:36.521 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:36.522 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:36.699 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:36.699 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:36.715 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 678 kB/s | 145 kB 00:00 2024-03-29T11:30:36.716 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:36.716 INFO:teuthology.orchestra.run.smithi190.stdout:Total 674 kB/s | 145 kB 00:00 2024-03-29T11:30:36.716 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:36.724 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:36.724 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:36.776 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:36.776 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:36.917 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:36.975 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:37.000 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:37.000 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:37.000 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:30:37.000 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:37.219 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:37.306 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:30:37.341 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:30:37.389 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:30:37.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:30:37.518 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:30:37.648 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:30:37.749 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:30:37.799 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:30:37.827 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:37.859 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:38.006 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:38.114 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:30:38.150 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:38.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-03-29T11:30:38.802 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:38.890 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:38.890 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:30:38.890 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:30:38.890 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:30:38.891 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:30:38.899 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:38.951 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:38.952 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:30:38.952 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:39.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:30:39.277 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:30:39.278 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:30:39.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:30:39.279 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:39.279 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:39.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:30:39.578 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:39.730 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:39.731 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:30:39.732 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-29T11:30:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.9 M 2024-03-29T11:30:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-29T11:30:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:40.115 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:40.273 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:40.274 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:30:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-29T11:30:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:40.277 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2024-03-29T11:30:40.277 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 209 M 2024-03-29T11:30:40.277 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:40.504 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-29T11:30:40.554 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-03-29T11:30:40.578 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 143 kB/s | 31 kB 00:00 2024-03-29T11:30:40.638 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 410 kB/s | 34 kB 00:00 2024-03-29T11:30:40.795 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-29T11:30:40.989 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-03-29T11:30:41.004 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-03-29T11:30:41.029 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 238 kB 00:00 2024-03-29T11:30:41.321 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.9 MB/s | 1.8 MB 00:00 2024-03-29T11:30:41.421 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T11:30:41.447 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-29T11:30:41.597 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 964 kB/s | 144 kB 00:00 2024-03-29T11:30:41.639 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.4 MB/s | 2.0 MB 00:00 2024-03-29T11:30:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 459 kB/s | 31 kB 00:00 2024-03-29T11:30:41.698 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-03-29T11:30:41.715 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-29T11:30:41.717 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:41.717 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.5 MB/s | 6.9 MB 00:01 2024-03-29T11:30:41.736 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:41.753 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:41.753 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:41.941 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:41.941 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:42.164 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-14 4.1 MB/s | 7.4 MB 00:01 2024-03-29T11:30:42.381 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 2.7 MB/s | 4.7 MB 00:01 2024-03-29T11:30:42.456 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:42.523 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.1 MB/s | 4.8 MB 00:01 2024-03-29T11:30:42.540 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:30:42.548 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 566 kB/s | 217 kB 00:00 2024-03-29T11:30:42.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:30:42.631 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:30:42.661 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:30:42.731 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T11:30:42.762 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:30:42.857 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-29T11:30:42.889 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:30:42.991 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:30:43.041 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:30:43.069 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:43.101 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:30:43.165 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 567 kB/s | 245 kB 00:00 2024-03-29T11:30:43.348 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:30:43.385 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:43.741 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-03-29T11:30:43.743 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:43.744 INFO:teuthology.orchestra.run.smithi045.stdout:Total 11 MB/s | 37 MB 00:03 2024-03-29T11:30:43.788 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:43.815 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:43.815 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:44.170 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:30:44.171 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:30:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:30:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:30:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:30:44.172 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:30:44.195 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:44.196 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:44.524 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:30:44.524 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:44.524 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:30:44.525 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:30:44.526 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:44.526 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:44.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:30:44.926 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:44.997 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:30:45.028 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:30:45.128 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:45.165 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:30:45.330 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:45.349 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:45.372 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:45.451 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:30:45.482 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:45.483 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:30:45.484 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-29T11:30:45.485 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:45.486 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2024-03-29T11:30:45.486 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 209 M 2024-03-29T11:30:45.486 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:45.755 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 287 kB/s | 32 kB 00:00 2024-03-29T11:30:45.814 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 485 kB/s | 17 kB 00:00 2024-03-29T11:30:45.856 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-29T11:30:46.106 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T11:30:46.151 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:30:46.273 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:30:46.290 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-29T11:30:46.340 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-03-29T11:30:46.365 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-29T11:30:46.532 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-03-29T11:30:46.566 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 6.1 MB/s | 195 kB 00:00 2024-03-29T11:30:46.842 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-29T11:30:47.008 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 556 kB/s | 245 kB 00:00 2024-03-29T11:30:47.564 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:30:47.858 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.3 MB/s | 7.4 MB 00:02 2024-03-29T11:30:47.860 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:47.861 INFO:teuthology.orchestra.run.smithi190.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T11:30:47.904 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:47.929 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:47.929 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:48.287 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:48.287 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:48.652 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:30:48.682 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:49.073 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:30:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:30:49.205 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:49.250 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:30:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:49.546 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:30:50.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:30:50.302 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:30:51.402 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:30:51.403 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:30:51.403 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:30:51.403 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:30:51.531 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:51.863 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:52.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:30:52.571 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:30:52.598 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:52.812 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:52.970 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:30:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:52.972 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-03-29T11:30:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:52.973 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2024-03-29T11:30:52.973 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-29T11:30:52.973 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:53.476 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-29T11:30:53.502 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-29T11:30:53.552 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:30:53.577 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:30:53.619 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-29T11:30:53.644 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 43 kB 00:00 2024-03-29T11:30:53.720 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 136 kB/s | 52 kB 00:00 2024-03-29T11:30:53.753 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-29T11:30:53.778 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-29T11:30:53.887 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-03-29T11:30:53.938 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 932 kB/s | 203 kB 00:00 2024-03-29T11:30:53.971 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-29T11:30:54.013 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T11:30:54.015 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:54.015 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-29T11:30:54.015 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:30:54.030 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:30:54.030 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:30:54.127 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:30:54.127 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:30:54.648 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:30:54.818 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:30:54.875 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:30:54.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:30:54.966 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:30:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:30:55.107 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:30:55.180 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:30:55.181 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:30:55.181 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:30:55.181 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:30:55.223 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:30:55.277 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:30:55.616 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:30:55.617 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:30:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:30:55.618 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:55.618 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:30:55.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:30:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:30:56.083 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:30:56.117 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:30:56.166 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:30:56.195 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:56.541 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:30:56.692 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:30:56.741 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:30:56.742 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-29T11:30:56.743 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:30:56.884 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 218 kB/s | 24 kB 00:00 2024-03-29T11:30:56.909 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2024-03-29T11:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:56.937 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:30:56.938 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:30:56.959 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:30:56.985 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:30:57.010 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:30:57.035 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:30:57.094 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 162 kB/s | 52 kB 00:00 2024-03-29T11:30:57.152 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:30:57.178 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:30:57.286 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:30:57.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:57.302 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:30:57.311 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 932 kB/s | 203 kB 00:00 2024-03-29T11:30:57.370 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T11:30:57.412 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T11:30:57.413 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:57.414 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T11:30:57.414 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:30:57.427 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:30:57.427 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:30:57.518 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:30:57.518 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:30:57.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:30:58.072 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:30:58.155 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:30:58.241 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:30:58.288 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:30:58.312 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:30:58.313 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:58.313 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:30:58.313 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:58.313 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:30:58.313 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-29T11:30:58.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:30:58.315 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.8 M 2024-03-29T11:30:58.315 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2024-03-29T11:30:58.315 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:30:58.344 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:30:58.381 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:30:58.422 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:30:58.524 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:30:58.577 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T11:30:58.633 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:30:58.682 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:30:58.860 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-29T11:30:58.935 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-29T11:30:58.961 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 576 kB/s | 58 kB 00:00 2024-03-29T11:30:58.986 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T11:30:59.398 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:30:59.446 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:30:59.480 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:30:59.528 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:30:59.550 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:31:00.281 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:31:00.282 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:31:00.283 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:31:00.636 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:00.637 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:00.836 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 57 kB/s | 138 kB 00:02 2024-03-29T11:31:00.839 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:00.839 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.7 MB/s | 6.8 MB 00:02 2024-03-29T11:31:00.855 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:00.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:31:00.868 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:00.868 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:00.964 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:00.965 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:01.276 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:01.435 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:01.452 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:31:01.514 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:31:01.590 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:01.590 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:31:01.590 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-29T11:31:01.592 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-29T11:31:01.592 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:01.592 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.8 M 2024-03-29T11:31:01.592 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-03-29T11:31:01.592 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:02.076 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:31:02.161 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:31:02.223 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-29T11:31:02.230 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:31:02.248 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 602 kB/s | 248 kB 00:00 2024-03-29T11:31:02.259 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:31:02.306 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 293 kB/s | 138 kB 00:00 2024-03-29T11:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-03-29T11:31:02.507 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 254 kB/s | 51 kB 00:00 2024-03-29T11:31:02.799 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 952 kB/s | 548 kB 00:00 2024-03-29T11:31:02.801 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:02.802 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-29T11:31:02.817 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:02.828 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:02.829 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:02.919 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:02.919 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:31:02.997 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:31:03.217 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:31:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:03.318 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:03.371 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:31:03.418 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:31:03.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-03-29T11:31:03.934 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:31:04.020 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:31:04.081 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:31:04.104 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:31:04.147 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:04.303 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:31:04.304 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:04.305 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 964 k 2024-03-29T11:31:04.306 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.8 M 2024-03-29T11:31:04.306 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:04.480 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 451 kB/s | 79 kB 00:00 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:31:04.888 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:31:05.195 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:31:05.195 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:05.195 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:05.195 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:05.195 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:05.196 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:05.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-03-29T11:31:05.538 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 718 kB/s | 885 kB 00:01 2024-03-29T11:31:05.540 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:05.540 INFO:teuthology.orchestra.run.smithi045.stdout:Total 780 kB/s | 964 kB 00:01 2024-03-29T11:31:05.540 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:05.565 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:05.565 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:05.645 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:05.646 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:05.833 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:05.969 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:31:06.020 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:06.041 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:31:06.171 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:06.172 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 964 k 2024-03-29T11:31:06.173 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.8 M 2024-03-29T11:31:06.173 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:06.301 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 613 kB/s | 79 kB 00:00 2024-03-29T11:31:06.651 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 885 kB 00:00 2024-03-29T11:31:06.652 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:06.653 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-29T11:31:06.653 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:06.673 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:06.673 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:06.716 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:31:06.716 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:31:06.747 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:06.747 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:06.934 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:07.008 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:31:07.143 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:31:07.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-volume 2024-03-29T11:31:07.836 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:07.871 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:31:07.871 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:31:07.995 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:07.995 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-29T11:31:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:07.997 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 332 k 2024-03-29T11:31:07.997 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 1.5 M 2024-03-29T11:31:07.997 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:08.128 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 526 kB/s | 69 kB 00:00 2024-03-29T11:31:08.201 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:31:08.201 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:08.202 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:08.202 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:31:08.202 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:08.202 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:08.478 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 548 kB/s | 264 kB 00:00 2024-03-29T11:31:08.480 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:08.480 INFO:teuthology.orchestra.run.smithi045.stdout:Total 687 kB/s | 332 kB 00:00 2024-03-29T11:31:08.480 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:08.490 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:08.490 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:08.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-03-29T11:31:08.529 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:08.530 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:08.684 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:08.876 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:31:08.906 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:31:08.922 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:31:08.922 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:08.922 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:31:08.922 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:09.084 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:09.237 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:09.237 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:09.237 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-29T11:31:09.237 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:09.237 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-29T11:31:09.238 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:09.239 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 332 k 2024-03-29T11:31:09.239 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.5 M 2024-03-29T11:31:09.239 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:09.375 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 506 kB/s | 69 kB 00:00 2024-03-29T11:31:09.606 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:31:09.642 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 655 kB/s | 264 kB 00:00 2024-03-29T11:31:09.642 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:09.643 INFO:teuthology.orchestra.run.smithi190.stdout:Total 821 kB/s | 332 kB 00:00 2024-03-29T11:31:09.643 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:09.651 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:09.651 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:09.690 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:09.691 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:09.835 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:09.950 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:31:09.950 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:09.950 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:09.950 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:09.950 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:31:09.951 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:09.951 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:10.023 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:31:10.061 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:31:10.077 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:31:10.077 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:10.077 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:31:10.077 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:10.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-03-29T11:31:10.801 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:31:10.873 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:11.038 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:31:11.038 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:11.038 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 127 k 2024-03-29T11:31:11.042 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 456 k 2024-03-29T11:31:11.042 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:11.162 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:11.411 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 340 kB/s | 127 kB 00:00 2024-03-29T11:31:11.412 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:11.412 INFO:teuthology.orchestra.run.smithi045.stdout:Total 338 kB/s | 127 kB 00:00 2024-03-29T11:31:11.412 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:11.418 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:11.418 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:11.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-03-29T11:31:11.476 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:11.476 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:11.628 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:11.738 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:12.041 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:12.173 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:12.193 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:31:12.194 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:12.195 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-03-29T11:31:12.195 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-03-29T11:31:12.195 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:12.471 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:12.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-03-29T11:31:12.806 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 207 kB/s | 127 kB 00:00 2024-03-29T11:31:12.807 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:12.807 INFO:teuthology.orchestra.run.smithi190.stdout:Total 207 kB/s | 127 kB 00:00 2024-03-29T11:31:12.807 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:12.813 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:12.813 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:12.868 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:12.869 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:13.015 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:13.123 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:13.273 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:13.375 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:13.432 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:13.433 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:13.433 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:13.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-03-29T11:31:13.542 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:13.818 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:13.819 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:14.075 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:14.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-03-29T11:31:14.232 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:14.233 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:14.233 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T11:31:14.233 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 31 k 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 140 k 2024-03-29T11:31:14.234 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:14.633 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 78 kB/s | 31 kB 00:00 2024-03-29T11:31:14.633 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:14.634 INFO:teuthology.orchestra.run.smithi045.stdout:Total 78 kB/s | 31 kB 00:00 2024-03-29T11:31:14.634 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:14.637 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:14.637 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:14.660 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:14.662 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:14.662 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:14.763 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:14.794 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:14.819 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:14.820 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:14.821 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:14.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-03-29T11:31:14.892 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:15.454 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:15.531 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:15.532 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:15.532 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:15.609 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:15.610 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:15.611 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:31:15.611 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:15.611 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 k 2024-03-29T11:31:15.611 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 140 k 2024-03-29T11:31:15.611 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:15.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-03-29T11:31:16.094 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 64 kB/s | 31 kB 00:00 2024-03-29T11:31:16.095 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:16.095 INFO:teuthology.orchestra.run.smithi190.stdout:Total 64 kB/s | 31 kB 00:00 2024-03-29T11:31:16.095 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:16.098 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:16.098 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:16.143 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:16.143 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:16.262 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:16.269 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:16.369 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:16.424 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:16.425 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:16.425 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:16.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-03-29T11:31:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:17.047 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:17.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-03-29T11:31:17.203 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:17.204 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:17.204 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:17.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-03-29T11:31:17.720 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:17.819 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:17.831 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:17.871 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:17.872 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:17.872 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:17.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-03-29T11:31:17.933 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:17.989 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:17.990 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:17.990 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:18.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-03-29T11:31:18.482 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:18.581 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:18.619 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:18.634 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:18.635 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:18.635 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:18.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-03-29T11:31:18.720 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:18.775 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:18.776 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:18.777 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:18.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-03-29T11:31:19.258 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:19.358 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:19.410 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:19.411 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:19.411 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:19.418 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:19.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-03-29T11:31:19.521 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:19.576 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:19.577 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:19.577 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:19.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-03-29T11:31:20.028 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:20.127 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:20.179 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:20.180 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:20.180 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:20.217 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:20.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-03-29T11:31:20.318 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:20.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-03-29T11:31:20.799 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:20.898 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:20.950 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:20.951 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:20.951 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:20.989 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:21.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-03-29T11:31:21.145 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 85 k 2024-03-29T11:31:21.146 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 231 k 2024-03-29T11:31:21.147 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:21.571 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:21.653 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 168 kB/s | 85 kB 00:00 2024-03-29T11:31:21.653 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:21.654 INFO:teuthology.orchestra.run.smithi045.stdout:Total 168 kB/s | 85 kB 00:00 2024-03-29T11:31:21.654 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:21.662 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:21.662 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:21.669 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:21.721 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:21.721 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:21.722 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-29T11:31:21.722 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:21.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-03-29T11:31:21.981 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:22.079 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:22.339 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:22.488 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:22.488 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:31:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:22.490 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 85 k 2024-03-29T11:31:22.490 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 231 k 2024-03-29T11:31:22.490 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:22.597 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:22.897 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:23.080 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 144 kB/s | 85 kB 00:00 2024-03-29T11:31:23.081 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:23.081 INFO:teuthology.orchestra.run.smithi190.stdout:Total 144 kB/s | 85 kB 00:00 2024-03-29T11:31:23.081 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:23.088 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:23.088 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:23.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-03-29T11:31:23.144 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:23.144 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:23.723 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:23.855 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:23.881 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:23.882 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:23.882 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 13 M 2024-03-29T11:31:23.883 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:24.177 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:24.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:24.178 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:24.178 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.178 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:24.178 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:24.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-03-29T11:31:24.993 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:25.146 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-03-29T11:31:25.147 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 13 M 2024-03-29T11:31:25.148 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:25.634 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:31:25.635 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:25.635 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:31:25.635 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:25.646 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:25.647 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:25.728 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:25.729 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:26.188 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:26.250 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:26.275 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:26.275 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:26.275 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:31:26.276 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:26.276 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:26.276 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:26.634 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.1 MB/s | 3.1 MB 00:01 2024-03-29T11:31:26.635 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:26.636 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.1 MB/s | 3.1 MB 00:01 2024-03-29T11:31:26.636 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:26.647 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:26.647 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:26.724 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:26.724 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:27.157 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:27.218 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:27.244 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:31:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:27.245 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:27.454 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:27.454 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:27.454 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:27.454 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:27.454 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:27.455 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:27.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-03-29T11:31:28.250 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:29:19 AM UTC. 2024-03-29T11:31:28.380 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:28.380 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:28.380 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:28.380 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:28.380 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:28.381 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:28.406 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-29T11:31:28.407 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:28.407 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:28.407 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-29T11:31:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 171 k 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 494 k 2024-03-29T11:31:28.408 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-29T11:31:28.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-03-29T11:31:29.163 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:29:39 AM UTC. 2024-03-29T11:31:29.203 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 216 kB/s | 171 kB 00:00 2024-03-29T11:31:29.204 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:Total 215 kB/s | 171 kB 00:00 2024-03-29T11:31:29.205 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-29T11:31:29.214 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-29T11:31:29.214 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-29T11:31:29.273 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-29T11:31:29.273 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-29T11:31:29.313 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-29T11:31:29.314 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:29.314 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 171 k 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 494 k 2024-03-29T11:31:29.315 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-29T11:31:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-29T11:31:29.530 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:29.965 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:30.047 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 234 kB/s | 171 kB 00:00 2024-03-29T11:31:30.048 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:30.048 INFO:teuthology.orchestra.run.smithi190.stdout:Total 234 kB/s | 171 kB 00:00 2024-03-29T11:31:30.048 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-29T11:31:30.056 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-29T11:31:30.056 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-29T11:31:30.112 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-29T11:31:30.113 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:30.254 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-29T11:31:30.269 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-29T11:31:30.378 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:30.533 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:30.810 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:31.072 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-29T11:31:31.271 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:31.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:31.415 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:31:31.439 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:31.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:31.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:31:31.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:31.581 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:31:31.602 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:31.603 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:31.603 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:31:31.605 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:31:31.605 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:31.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:31:31.637 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:31.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:31:31.670 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:31:31.671 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:31.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:31:31.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:31:31.778 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:31.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:31:31.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:31:31.872 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:31:31.872 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:31.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:31:31.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:31:31.972 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:31.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:31:31.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:31:32.065 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:31:32.066 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:32.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:31:32.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:31:32.164 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:32.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:31:32.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:31:32.259 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:31:32.351 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:31:32.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-03-29T11:31:32.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-03-29T11:31:32.381 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:31:32.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:31:32.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:31:32.450 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:31:32.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:31:32.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:31:32.530 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:31:32.530 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:31:32.531 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622724', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTYl8mD0g8ZLBN36uwr+/8pyQ1eLH6j5qG+Sx0y7nd16YU3Vam4ldskPPXd7k656VJQ76Tk/gP4GQaHQDxxtpI=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCyO5V/BD/RsJnRH35tRBrBritxo2PqbBiQAiauzZgHMv0s0pwjuu5tEwE6RvCj+d/pKhHmHH9wYy3Nqxd1u0U0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:31:32.531 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:31:32.566 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:31:32.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:31:32.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:31:32.654 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:32.654 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:31:32.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:31:32.705 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-03-29T11:31:32.761 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:31:32.761 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:32.761 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:31:32.761 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:32.761 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:32.762 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 11:31:26.545425956 +0000 2024-03-29T11:31:32.762 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 11:28:36.318634298 +0000 2024-03-29T11:31:32.762 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 11:28:36.318634298 +0000 2024-03-29T11:31:32.762 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 11:28:36.318634298 +0000 2024-03-29T11:31:32.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:31:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T11:31:32.828 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T11:31:32.829 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000103958 s, 4.9 MB/s 2024-03-29T11:31:32.830 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:31:32.886 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-03-29T11:31:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 11:31:26.545425956 +0000 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 11:28:36.653629117 +0000 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 11:28:36.653629117 +0000 2024-03-29T11:31:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 11:28:36.653629117 +0000 2024-03-29T11:31:32.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T11:31:33.008 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T11:31:33.009 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.7693e-05 s, 5.2 MB/s 2024-03-29T11:31:33.010 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:31:33.066 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 11:31:26.545425956 +0000 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 11:28:36.978624090 +0000 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 11:28:36.978624090 +0000 2024-03-29T11:31:33.121 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 11:28:36.978624090 +0000 2024-03-29T11:31:33.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:31:33.187 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T11:31:33.187 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T11:31:33.187 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 7.9756e-05 s, 6.4 MB/s 2024-03-29T11:31:33.188 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:31:33.244 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-29 11:31:26.546425941 +0000 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-29 11:28:37.309618971 +0000 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-29 11:28:37.309618971 +0000 2024-03-29T11:31:33.299 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-29 11:28:37.309618971 +0000 2024-03-29T11:31:33.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:31:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-29T11:31:33.365 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-29T11:31:33.366 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.6012e-05 s, 5.3 MB/s 2024-03-29T11:31:33.367 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:31:33.426 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:33.426 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:33.426 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:31:33.445 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:31:33.445 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-03-29T11:31:33.502 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:31:33.502 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:33.502 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-29 11:31:27.514817534 +0000 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-29 11:28:31.213624440 +0000 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-29 11:28:31.213624440 +0000 2024-03-29T11:31:33.503 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-29 11:28:31.213624440 +0000 2024-03-29T11:31:33.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:31:33.572 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-29T11:31:33.572 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-29T11:31:33.572 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000217149 s, 2.4 MB/s 2024-03-29T11:31:33.573 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:31:33.631 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-03-29T11:31:33.687 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:31:33.687 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:33.687 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-29 11:31:27.515817507 +0000 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-29 11:28:31.556614777 +0000 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-29 11:28:31.556614777 +0000 2024-03-29T11:31:33.688 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-29 11:28:31.556614777 +0000 2024-03-29T11:31:33.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:31:33.755 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-29T11:31:33.755 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-29T11:31:33.755 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000212825 s, 2.4 MB/s 2024-03-29T11:31:33.756 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:31:33.813 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-03-29T11:31:33.869 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-29 11:31:27.515817507 +0000 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-29 11:28:31.891605340 +0000 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-29 11:28:31.891605340 +0000 2024-03-29T11:31:33.870 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-29 11:28:31.891605340 +0000 2024-03-29T11:31:33.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:31:33.939 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-29T11:31:33.939 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-29T11:31:33.939 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00024308 s, 2.1 MB/s 2024-03-29T11:31:33.941 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:31:33.997 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:34.052 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-29 11:31:27.515817507 +0000 2024-03-29T11:31:34.053 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-29 11:28:32.221596044 +0000 2024-03-29T11:31:34.053 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-29 11:28:32.221596044 +0000 2024-03-29T11:31:34.053 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-29 11:28:32.221596044 +0000 2024-03-29T11:31:34.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:31:34.118 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-29T11:31:34.118 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-29T11:31:34.118 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000217815 s, 2.4 MB/s 2024-03-29T11:31:34.119 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:31:34.175 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:34.176 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:31:34.176 INFO:tasks.ceph:Generating config... 2024-03-29T11:31:34.177 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:31:34.177 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:31:34.177 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:31:34.177 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:31:34.177 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:31:34.177 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:31:34.178 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:31:34.178 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:31:34.178 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:31:34.178 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:31:34.178 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:31:34.179 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:31:34.180 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:31:34.180 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:31:34.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:31:34.233 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:31:34.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:31:34.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:34.311 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.45:6789]'), ('mon.c', '[v1:172.21.15.45:6790]'), ('mon.b', '[v1:172.21.15.190:6789]')] 2024-03-29T11:31:34.311 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.190:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:31:34.312 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:34.312 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:31:34.366 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.45:6789]' --addv c '[v1:172.21.15.45:6790]' --addv b '[v1:172.21.15.190:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:34.451 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 17 (quincy) 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:31:34.452 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:31:34.454 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:31:34.468 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5... 2024-03-29T11:31:34.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:31:34.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-03-29T11:31:34.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-03-29T11:31:34.560 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.561 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.190:6789]" 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: lockdep = True 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: mon client directed command retry = 5 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: mon election default strategy = 1 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: ms bind msgr2 = False 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject socket failures = 5000 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: ms type = async 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:31:34.562 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-03-29T11:31:34.563 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-03-29T11:31:34.564 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression algorithm = snappy 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression mode = aggressive 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore zero block detection = True 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-03-29T11:31:34.565 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd class default list = * 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd class load list = * 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd client watch timeout = 120 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify cached snaps = True 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify missing on start = True 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock override recovery settings = True 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue = debug_random 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue cut off = debug_random 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: osd shutdown pgref assert = True 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.566 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-03-29T11:31:34.567 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.568 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon min osdmap epochs = 50 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune interval = 2 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune min = 15 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub interval = 300 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on pool no app = False 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: paxos service trim min = 10 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-03-29T11:31:34.569 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug objecter = 20 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout: debug rados = 20 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-03-29T11:31:34.570 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:31:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "[v1:172.21.15.45:6789],[v1:172.21.15.45:6790],[v1:172.21.15.190:6789]" 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: lockdep = True 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 1 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: ms bind msgr2 = False 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: ms type = async 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-03-29T11:31:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-03-29T11:31:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression algorithm = snappy 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression mode = aggressive 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: osd class default list = * 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: osd class load list = * 2024-03-29T11:31:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: osd client watch timeout = 120 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd shutdown pgref assert = True 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-03-29T11:31:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-03-29T11:31:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:31:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: mon min osdmap epochs = 50 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune interval = 2 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune min = 15 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: paxos service trim min = 10 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:31:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug objecter = 20 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug rados = 20 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-03-29T11:31:34.582 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-03-29T11:31:34.583 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:31:34.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:31:34.678 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:31:34.678 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:34.678 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:31:34.693 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:34.693 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:31:34.747 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:31:34.748 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:34.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:31:34.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:34.827 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:31:34.827 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:34.882 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:31:34.882 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:34.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:31:34.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:34.923 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:31:34.923 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:34.978 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:31:34.978 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:31:34.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:31:35.032 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:31:35.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:31:35.096 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:31:35.098 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:31:35.099 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:31:35.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:31:35.141 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:31:35.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:31:35.198 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:31:35.213 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:31:35.213 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:31:35.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:35.241 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:35.241 INFO:tasks.ceph:role: osd.0 2024-03-29T11:31:35.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:31:35.241 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:35.308 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:35.308 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:35.308 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:35.339 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T11:31:35.341 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T11:31:35.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:35.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:35.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:35.550 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:35.551 INFO:tasks.ceph:role: osd.1 2024-03-29T11:31:35.551 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:31:35.551 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:35.617 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:35.618 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:35.618 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:35.650 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T11:31:35.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T11:31:35.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:35.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:35.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:35.860 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:35.861 INFO:tasks.ceph:role: osd.2 2024-03-29T11:31:35.861 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:31:35.861 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:35.929 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:35.930 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:35.961 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T11:31:35.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T11:31:35.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:36.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:36.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:36.175 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:36.175 INFO:tasks.ceph:role: osd.3 2024-03-29T11:31:36.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:31:36.176 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:31:36.241 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:36.242 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:36.274 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-03-29T11:31:36.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-03-29T11:31:36.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:36.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:36.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.522+0000 7f19d280e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.523+0000 7f19d280e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.523+0000 7f19d280e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.523+0000 7f19d280e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.523+0000 7f19d280e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:36.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:36.523+0000 7f19d280e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:31:39.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:39.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:39.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:39.492+0000 7f34ff264640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:31:42.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:42.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.474+0000 7f33dbb49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:31:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.474+0000 7f33dbb49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:31:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.474+0000 7f33dbb49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:42.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.474+0000 7f33dbb49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.475+0000 7f33dbb49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:42.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:42.475+0000 7f33dbb49640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:31:45.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:45.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:45.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:31:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:31:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:45.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:31:45.447+0000 7efcbc1fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:31:48.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:48.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:48.354 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:48.354 INFO:tasks.ceph:role: osd.4 2024-03-29T11:31:48.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:31:48.354 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:48.432 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:48.433 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:48.433 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:48.954 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-29T11:31:48.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-29T11:31:48.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:49.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:49.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:49.222 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:49.222 INFO:tasks.ceph:role: osd.5 2024-03-29T11:31:49.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:31:49.223 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:31:49.298 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:49.298 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:49.746 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-29T11:31:49.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-29T11:31:49.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:49.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:49.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:49.962 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:49.962 INFO:tasks.ceph:role: osd.6 2024-03-29T11:31:49.962 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:31:49.962 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:50.037 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:50.523 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-29T11:31:50.525 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-29T11:31:50.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:50.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:50.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:50.768 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:50.768 INFO:tasks.ceph:role: osd.7 2024-03-29T11:31:50.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:31:50.768 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:31:50.842 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:50.843 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:51.351 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-29T11:31:51.354 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-29T11:31:51.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:51.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:51.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:51.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.661+0000 7f2c84d06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:31:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.662+0000 7f2c84d06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:31:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.662+0000 7f2c84d06640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.662+0000 7f2c84d06640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.662+0000 7f2c84d06640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:51.662+0000 7f2c84d06640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:31:54.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:54.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:54.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.629+0000 7efda4e8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:31:54.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.629+0000 7efda4e8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:31:54.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.629+0000 7efda4e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:54.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.629+0000 7efda4e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:54.752 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.629+0000 7efda4e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:54.752 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:54.630+0000 7efda4e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:31:57.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:57.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.592+0000 7f4cb4a0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.593+0000 7f4cb4a0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.593+0000 7f4cb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.593+0000 7f4cb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.593+0000 7f4cb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:31:57.593+0000 7f4cb4a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:32:00.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:32:00.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:32:00.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:32:00.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:32:00.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:32:00.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:32:00.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:32:00.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:00.566+0000 7f82bee0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:32:03.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:32:03.440 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:32:03.440 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:03.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:32:03.473 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:03.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:32:03.543 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:03.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:32:03.613 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:03.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:32:03.683 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:03.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:32:03.753 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:03.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:32:03.784 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:03.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:32:03.854 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:03.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:32:03.924 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:03.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:32:03.992 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:03.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:32:04.061 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:04.061 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:32:04.079 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:04.079 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:32:04.116 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:32:04.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:32:04.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:32:04.159 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCXpgZmFUhdAhAAYpe/oo7ASxZZbKi8R4Hcbg== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCYpgZmWxQ+HxAAHZoa5VlWW0KlK4Pw8iX04A== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCbpgZmqjJnHRAAzbTKe1AL+7gDq75Xy7V5Hg== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCepgZmfEpeHBAAN97vJUS2PHhs95j/J1PysQ== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQChpgZmhuOsGhAArSMzhcnybDawl7sgd8pZ2g== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCXpgZm87wpBhAAO4sfh48Kw0bs8BlrxPkW8A== 2024-03-29T11:32:04.160 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCnpgZmF1ODJxAA8ZlAN13WcGgXLKQHJQSh9Q== 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCqpgZmMjCTJRAAAb6tSyJtrnRvnzzYOYBNcA== 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCtpgZmsmZ2IxAAXupwWlib6/s3rSWWdqkbYg== 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCwpgZmNafLIRAAqdPj35U/MtFtgM1vraF+Hg== 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCXpgZmJonRCBAAKGPBWOpnabqX0kKanaosog== 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2024-03-29T11:32:04.161 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQCXpgZmknQ9DBAAego2LfmD++9IxD9PNoU8rQ== 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.y] 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCXpgZmFUhdAhAAYpe/oo7ASxZZbKi8R4Hcbg== 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCYpgZmWxQ+HxAAHZoa5VlWW0KlK4Pw8iX04A== 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-03-29T11:32:04.182 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCbpgZmqjJnHRAAzbTKe1AL+7gDq75Xy7V5Hg== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCepgZmfEpeHBAAN97vJUS2PHhs95j/J1PysQ== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQChpgZmhuOsGhAArSMzhcnybDawl7sgd8pZ2g== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCXpgZm87wpBhAAO4sfh48Kw0bs8BlrxPkW8A== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCnpgZmF1ODJxAA8ZlAN13WcGgXLKQHJQSh9Q== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCqpgZmMjCTJRAAAb6tSyJtrnRvnzzYOYBNcA== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.6] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCtpgZmsmZ2IxAAXupwWlib6/s3rSWWdqkbYg== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.7] 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCwpgZmNafLIRAAqdPj35U/MtFtgM1vraF+Hg== 2024-03-29T11:32:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-03-29T11:32:04.184 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCXpgZmJonRCBAAKGPBWOpnabqX0kKanaosog== 2024-03-29T11:32:04.184 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-03-29T11:32:04.184 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCXpgZmknQ9DBAAego2LfmD++9IxD9PNoU8rQ== 2024-03-29T11:32:04.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:32:04.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:32:04.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:32:04.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:32:04.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:32:04.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:32:04.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:32:04.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:32:04.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:32:04.926 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:32:04.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:32:04.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:32:05.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:32:05.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:32:05.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:32:05.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:32:05.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:32:05.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:32:05.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:32:05.424 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:32:05.479 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:32:05.494 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:32:05.495 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:32:05.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:32:05.536 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:32:05.536 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:32:05.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:32:05.538 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:32:05.538 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:32:05.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:32:05.542 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:32:05.542 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:32:05.542 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:32:05.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:32:05.545 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:32:05.545 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:32:05.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:32:05.548 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:32:05.549 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:32:05.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.659+0000 7f184cfcf640 10 monclient: get_monmap_and_config 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.659+0000 7f184cfcf640 10 monclient: build_initial_monmap 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.659+0000 7f184cfcf640 10 monclient: monmap: 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:05.654 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:32:05.660819+0000 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:32:05.660819+0000 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:05.655 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.660+0000 7f184cfcf640 1 Processor -- start 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.660+0000 7f184cfcf640 1 -- start start 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.660+0000 7f184cfcf640 10 monclient: init 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.660+0000 7f184cfcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.660+0000 7f184cfcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:05.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-c con 0x7f18480b8240 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-b con 0x7f18480b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-a con 0x7f184809c8c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f183ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f18480b1f60 0x7f184809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49044/0 (socket says 172.21.15.45:49044) 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f183ffff640 1 -- 172.21.15.45:0/3423040680 learned_addr learned my addr 172.21.15.45:0/3423040680 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184809bd80 con 0x7f184809c8c0 2024-03-29T11:32:05.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184809b960 con 0x7f18480b8240 2024-03-29T11:32:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b2ec0 con 0x7f18480b1f60 2024-03-29T11:32:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:05.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:05.661+0000 7f184cfcf640 10 monclient(hunting): authenticate will time out at 781.900405s 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): tick 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 monclient(hunting): continuing hunt 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.45:6790/0 conn(0x7f18480b1f60 legacy=0x7f184809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.190:6789/0 conn(0x7f18480b8240 legacy=0x7f184813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.45:6789/0 conn(0x7f184809c8c0 legacy=0x7f184814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:08.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): picked mon.noname-a con 0x7f18280036e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): picked mon.noname-b con 0x7f1828007290 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): picked mon.noname-c con 0x7f18480b8240 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184813fb60 con 0x7f18280036e0 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182800f760 con 0x7f18480b8240 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280109c0 con 0x7f1828007290 2024-03-29T11:32:08.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:08.661+0000 7f1844d99640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827794559 0 0) 0x7f184813fb60 con 0x7f18280036e0 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:32:10.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.862+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834003520 con 0x7f18280036e0 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233815252 0 0) 0x7f1834003520 con 0x7f18280036e0 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.45:6790/0 conn(0x7f1828007290 legacy=0x7f18280096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.190:6789/0 conn(0x7f18480b8240 legacy=0x7f182800cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 monclient: found mon.noname-a 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:10.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828011c20 con 0x7f18280036e0 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: _finish_auth 0 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18380016d0 con 0x7f18280036e0 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f184cfcf640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f184cfcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: dump: 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.861 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f184cfcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18380018a0 con 0x7f18280036e0 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 1 -- 172.21.15.45:0/3423040680 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1838001c30 con 0x7f18280036e0 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:10.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f1845d9b640 10 monclient: dump: 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f184cfcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:10.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.863+0000 7f184cfcf640 4 set_mon_vals no callback set 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 10 monclient: shutdown 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 >> v1:172.21.15.45:6789/0 conn(0x7f18280036e0 legacy=0x7f1828005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 shutdown_connections 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 >> 172.21.15.45:0/3423040680 conn(0x7f1848066b00 msgr2=0x7f1848066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 shutdown_connections 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.864+0000 7f184cfcf640 1 -- 172.21.15.45:0/3423040680 wait complete. 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.865+0000 7f184cfcf640 10 monclient: build_initial_monmap 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.865+0000 7f184cfcf640 10 monclient: monmap: 2024-03-29T11:32:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.865+0000 7f184cfcf640 1 librados: starting msgr at 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.865+0000 7f184cfcf640 1 librados: starting objecter 2024-03-29T11:32:10.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 1 Processor -- start 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 1 -- start start 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 1 librados: setting wanted keys 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 1 librados: calling monclient init 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 10 monclient: init 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-b con 0x7f18280036e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.866+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1828007290 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 10 monclient(hunting): picked mon.noname-a con 0x7f18480b8240 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480ade70 con 0x7f18480b8240 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b1e90 con 0x7f18280036e0 2024-03-29T11:32:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1828007290 0x7f18481d7770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48012/0 (socket says 172.21.15.45:48012) 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183ffff640 1 -- 172.21.15.45:0/1736069754 learned_addr learned my addr 172.21.15.45:0/1736069754 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18481d7250 con 0x7f1828007290 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f184cfcf640 10 monclient(hunting): authenticate will time out at 787.106430s 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter ms_handle_connect 0x7f18280036e0 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter ms_handle_connect 0x7f18480b8240 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter ms_handle_connect 0x7f1828007290 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951873353 0 0) 0x7f18480ade70 con 0x7f18480b8240 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.867+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18280100b0 con 0x7f18480b8240 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47891456 0 0) 0x7f18481d7250 con 0x7f1828007290 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18480ade70 con 0x7f1828007290 2024-03-29T11:32:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.868+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491554190 0 0) 0x7f18480b1e90 con 0x7f18280036e0 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18481d7250 con 0x7f18280036e0 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997164231 0 0) 0x7f18480ade70 con 0x7f1828007290 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18480b1e90 con 0x7f1828007290 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1830001830 con 0x7f1828007290 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): dump: 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:10.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392462448 0 0) 0x7f18481d7250 con 0x7f18280036e0 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18480ade70 con 0x7f18280036e0 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1838001bb0 con 0x7f18280036e0 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): dump: 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210856987 0 0) 0x7f18280100b0 con 0x7f18480b8240 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18481d7250 con 0x7f18480b8240 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1840001b40 con 0x7f18480b8240 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient(hunting): dump: 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1355127770 0 0) 0x7f18480ade70 con 0x7f18280036e0 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 >> v1:172.21.15.45:6790/0 conn(0x7f1828007290 legacy=0x7f18481d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 >> v1:172.21.15.45:6789/0 conn(0x7f18480b8240 legacy=0x7f184813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 monclient: found mon.b 2024-03-29T11:32:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18481d6e70 con 0x7f18280036e0 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient: _finish_auth 0 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f183effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f184cfcf640 5 monclient: authenticate success, global_id 4102 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f184cfcf640 10 monclient: _renew_subs 2024-03-29T11:32:10.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f184cfcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.869+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18480b13f0 con 0x7f18280036e0 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184cfcf640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184cfcf640 10 monclient: _renew_subs 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184cfcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f184813e990 con 0x7f18280036e0 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184cfcf640 1 librados: init done 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1838001d10 con 0x7f18280036e0 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1838003a40 con 0x7f18280036e0 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f184559a640 4 set_mon_vals no callback set 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 10 monclient: dump: 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.870+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1838003f30 con 0x7f18280036e0 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 612087335 0 0) 0x7f18380051f0 con 0x7f18280036e0 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f183effd640 10 client.4102.objecter ms_dispatch 0x7f1848000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f183effd640 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f183effd640 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f183effd640 20 client.4102.objecter dump_active .. 0 homeless 2024-03-29T11:32:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f184cfcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f184cfcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f184cfcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.871+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c005080 con 0x7f18280036e0 2024-03-29T11:32:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.878+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1838004280 con 0x7f18280036e0 2024-03-29T11:32:10.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.878+0000 7f183effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:10.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.878+0000 7f183effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:10.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.947+0000 7f184cfcf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:32:10.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.947+0000 7f184cfcf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:32:10.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.947+0000 7f184cfcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:10.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.947+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f182c005330 con 0x7f18280036e0 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.954+0000 7f183effd640 1 -- 172.21.15.45:0/1736069754 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f18380044e0 con 0x7f18280036e0 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.954+0000 7f183effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.954+0000 7f183effd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 10 librados: watch_flush enter 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 10 librados: watch_flush exit 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-29T11:32:10.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 10 client.4102.objecter successfully canceled tick 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 10 monclient: shutdown 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 >> v1:172.21.15.190:6789/0 conn(0x7f18280036e0 legacy=0x7f184813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 shutdown_connections 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 >> 172.21.15.45:0/1736069754 conn(0x7f1848066b00 msgr2=0x7f18480b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 shutdown_connections 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 -- 172.21.15.45:0/1736069754 wait complete. 2024-03-29T11:32:10.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:10.955+0000 7f184cfcf640 1 librados: shutdown 2024-03-29T11:32:10.963 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:32:10.963 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:32:10.964 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:10.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:32:10.995 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:10.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:32:11.066 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:11.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:32:11.135 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-29T11:32:11.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:32:11.202 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:11.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:32:11.231 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:11.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:32:11.301 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:11.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:32:11.342 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:11.347+0000 7f2bbb020040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:32:11.368 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:11.373+0000 7fe430cc7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:32:11.371 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-29T11:32:11.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:32:11.401 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:11.405+0000 7f2bbb020040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:32:11.424 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:11.429+0000 7fe430cc7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:32:11.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6 0 2024-03-29T11:32:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.572+0000 7f2646893640 10 monclient: get_monmap_and_config 2024-03-29T11:32:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.572+0000 7f2646893640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.573+0000 7f2646893640 10 monclient: monmap: 2024-03-29T11:32:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:11.573959+0000 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:11.573959+0000 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.573+0000 7f2646893640 1 Processor -- start 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.573+0000 7f2646893640 1 -- start start 2024-03-29T11:32:11.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.573+0000 7f2646893640 10 monclient: init 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient(hunting): picked mon.noname-c con 0x7f264009ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient(hunting): picked mon.noname-a con 0x7f264009c870 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient(hunting): picked mon.noname-b con 0x7f26400b4360 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264009dec0 con 0x7f264009c870 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264009da30 con 0x7f264009ac40 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400b39c0 con 0x7f26400b4360 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f2646893640 10 monclient(hunting): authenticate will time out at 776.887455s 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263ffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f264009ac40 0x7f26400b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35668/0 (socket says 172.21.15.190:35668) 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263ffff640 1 -- 172.21.15.190:0/1216201706 learned_addr learned my addr 172.21.15.190:0/1216201706 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583749477 0 0) 0x7f264009da30 con 0x7f264009ac40 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.574+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628003520 con 0x7f264009ac40 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797191034 0 0) 0x7f2628003520 con 0x7f264009ac40 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 >> v1:172.21.15.45:6790/0 conn(0x7f26400b4360 legacy=0x7f264014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 >> v1:172.21.15.45:6789/0 conn(0x7f264009c870 legacy=0x7f264013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 monclient: found mon.noname-c 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400b5160 con 0x7f264009ac40 2024-03-29T11:32:11.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2634001690 con 0x7f264009ac40 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: dump: 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.583 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2634001870 con 0x7f264009ac40 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 1 -- 172.21.15.190:0/1216201706 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2634001cc0 con 0x7f264009ac40 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:11.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 4 set_mon_vals no callback set 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f263effd640 10 monclient: dump: 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.585 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 10 monclient: shutdown 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 1 -- 172.21.15.190:0/1216201706 >> v1:172.21.15.190:6789/0 conn(0x7f264009ac40 legacy=0x7f26400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 1 -- 172.21.15.190:0/1216201706 shutdown_connections 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 1 -- 172.21.15.190:0/1216201706 >> 172.21.15.190:0/1216201706 conn(0x7f2640066b00 msgr2=0x7f2640066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 1 -- 172.21.15.190:0/1216201706 shutdown_connections 2024-03-29T11:32:11.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.575+0000 7f2646893640 1 -- 172.21.15.190:0/1216201706 wait complete. 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 10 monclient: monmap: 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 librados: starting msgr at 2024-03-29T11:32:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 librados: starting objecter 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 Processor -- start 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 -- start start 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 librados: setting wanted keys 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 1 librados: calling monclient init 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.576+0000 7f2646893640 10 monclient: init 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient(hunting): picked mon.noname-c con 0x7f264009ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient(hunting): picked mon.noname-b con 0x7f264013a630 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient(hunting): picked mon.noname-a con 0x7f2640142c70 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26401401b0 con 0x7f2640142c70 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2640140640 con 0x7f264013a630 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2640142200 con 0x7f264009ac40 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f2646893640 10 monclient(hunting): authenticate will time out at 776.890443s 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263f7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f264013a630 0x7f26401d71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35680/0 (socket says 172.21.15.190:35680) 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263f7fe640 1 -- 172.21.15.190:0/2872913437 learned_addr learned my addr 172.21.15.190:0/2872913437 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter ms_handle_connect 0x7f264013a630 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102431197 0 0) 0x7f2640140640 con 0x7f264013a630 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c004eb0 con 0x7f264013a630 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter ms_handle_connect 0x7f2640142c70 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter ms_handle_connect 0x7f264009ac40 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924937368 0 0) 0x7f2640142200 con 0x7f264009ac40 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2640140640 con 0x7f264009ac40 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991436058 0 0) 0x7f26401401b0 con 0x7f2640142c70 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2640142200 con 0x7f2640142c70 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766436779 0 0) 0x7f262c004eb0 con 0x7f264013a630 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26401401b0 con 0x7f264013a630 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f26300017c0 con 0x7f264013a630 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.577+0000 7f263cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient(hunting): dump: 2024-03-29T11:32:11.591 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2346570139 0 0) 0x7f26401401b0 con 0x7f264013a630 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 >> v1:172.21.15.45:6790/0 conn(0x7f264009ac40 legacy=0x7f2640139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 >> v1:172.21.15.45:6789/0 conn(0x7f2640142c70 legacy=0x7f26401e2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 monclient: found mon.b 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26401424e0 con 0x7f264013a630 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:32:11.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 10 monclient: _renew_subs 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26401d7900 con 0x7f264013a630 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 10 monclient: _renew_subs 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26401409e0 con 0x7f264013a630 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f2646893640 1 librados: init done 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2630001cb0 con 0x7f264013a630 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f26300037b0 con 0x7f264013a630 2024-03-29T11:32:11.594 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263e7fc640 4 set_mon_vals no callback set 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 monclient: dump: 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.595 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2630003d20 con 0x7f264013a630 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1172304305 0 0) 0x7f26300063d0 con 0x7f264013a630 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 10 client.4108.objecter ms_dispatch 0x7f2640000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:32:11.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.578+0000 7f263cff9640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:32:11.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.579+0000 7f2646893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.579+0000 7f2646893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.579+0000 7f2646893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.579+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620005080 con 0x7f264013a630 2024-03-29T11:32:11.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.581+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f26300041b0 con 0x7f264013a630 2024-03-29T11:32:11.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.581+0000 7f263cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.581+0000 7f263cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:11.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.652+0000 7f2646893640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6", "id": 0}] 2024-03-29T11:32:11.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.652+0000 7f2646893640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6", "id": 0}] 2024-03-29T11:32:11.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.652+0000 7f2646893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.652+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6", "id": 0} v 0) v1 -- 0x7f26200058b0 con 0x7f264013a630 2024-03-29T11:32:11.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f263cff9640 1 -- 172.21.15.190:0/2872913437 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2349078976 0 732707515) 0x7f26300057c0 con 0x7f264013a630 2024-03-29T11:32:11.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f263cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6", "id": 0}] 2024-03-29T11:32:11.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f263cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:11.653 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-03-29T11:32:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f2646893640 10 librados: watch_flush enter 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f2646893640 10 librados: watch_flush exit 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f2646893640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.658+0000 7f2646893640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 10 monclient: shutdown 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 >> v1:172.21.15.190:6789/0 conn(0x7f264013a630 legacy=0x7f26401d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 shutdown_connections 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 >> 172.21.15.190:0/2872913437 conn(0x7f2640066b00 msgr2=0x7f264013e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 shutdown_connections 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 -- 172.21.15.190:0/2872913437 wait complete. 2024-03-29T11:32:11.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.659+0000 7f2646893640 1 librados: shutdown 2024-03-29T11:32:11.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 4dc754de-2886-4a2f-8cd4-5c2501bfd966 1 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.772+0000 7ffa53425640 10 monclient: get_monmap_and_config 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.772+0000 7ffa53425640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.772+0000 7ffa53425640 10 monclient: monmap: 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:11.773828+0000 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:11.773828+0000 2024-03-29T11:32:11.769 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.773+0000 7ffa53425640 1 Processor -- start 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.773+0000 7ffa53425640 1 -- start start 2024-03-29T11:32:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.773+0000 7ffa53425640 10 monclient: init 2024-03-29T11:32:11.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c09ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c09c870 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c0b4360 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c09dec0 con 0x7ffa4c0b4360 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c09da30 con 0x7ffa4c09c870 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b39c0 con 0x7ffa4c09ac40 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 10 monclient(hunting): authenticate will time out at 777.087238s 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa509c0640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ffa4c09c870 0x7ffa4c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35694/0 (socket says 172.21.15.190:35694) 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa509c0640 1 -- 172.21.15.190:0/1947223380 learned_addr learned my addr 172.21.15.190:0/1947223380 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958142695 0 0) 0x7ffa4c09da30 con 0x7ffa4c09c870 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:32:11.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa38003520 con 0x7ffa4c09c870 2024-03-29T11:32:11.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299721011 0 0) 0x7ffa4c0b39c0 con 0x7ffa4c09ac40 2024-03-29T11:32:11.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c09da30 con 0x7ffa4c09ac40 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239482379 0 0) 0x7ffa440013c0 con 0x7ffa4c0b4360 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c0b39c0 con 0x7ffa4c0b4360 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721730237 0 0) 0x7ffa38003520 con 0x7ffa4c09c870 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 >> v1:172.21.15.45:6790/0 conn(0x7ffa4c09ac40 legacy=0x7ffa4c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 >> v1:172.21.15.45:6789/0 conn(0x7ffa4c0b4360 legacy=0x7ffa4c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 monclient: found mon.noname-c 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0b5160 con 0x7ffa4c09c870 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa48001690 con 0x7ffa4c09c870 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:32:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa43fff640 10 monclient: dump: 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.774+0000 7ffa53425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa48001870 con 0x7ffa4c09c870 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 1 -- 172.21.15.190:0/1947223380 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa48001c80 con 0x7ffa4c09c870 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 4 set_mon_vals no callback set 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa43fff640 10 monclient: dump: 2024-03-29T11:32:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 10 monclient: shutdown 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 1 -- 172.21.15.190:0/1947223380 >> v1:172.21.15.190:6789/0 conn(0x7ffa4c09c870 legacy=0x7ffa4c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 1 -- 172.21.15.190:0/1947223380 shutdown_connections 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 1 -- 172.21.15.190:0/1947223380 >> 172.21.15.190:0/1947223380 conn(0x7ffa4c066b00 msgr2=0x7ffa4c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 1 -- 172.21.15.190:0/1947223380 shutdown_connections 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.775+0000 7ffa53425640 1 -- 172.21.15.190:0/1947223380 wait complete. 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 10 monclient: monmap: 2024-03-29T11:32:11.796 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 librados: starting msgr at 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 librados: starting objecter 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 Processor -- start 2024-03-29T11:32:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 -- start start 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 librados: setting wanted keys 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 1 librados: calling monclient init 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.776+0000 7ffa53425640 10 monclient: init 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c09ac40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c0af970 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c139360 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1d7200 con 0x7ffa4c09ac40 2024-03-29T11:32:11.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b1a80 con 0x7ffa4c0af970 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0acd70 con 0x7ffa4c139360 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa53425640 10 monclient(hunting): authenticate will time out at 777.090260s 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa509c0640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ffa4c0af970 0x7ffa4c1d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35704/0 (socket says 172.21.15.190:35704) 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa509c0640 1 -- 172.21.15.190:0/1878078390 learned_addr learned my addr 172.21.15.190:0/1878078390 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter ms_handle_connect 0x7ffa4c0af970 2024-03-29T11:32:11.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter ms_handle_connect 0x7ffa4c09ac40 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter ms_handle_connect 0x7ffa4c139360 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743692924 0 0) 0x7ffa4c0b1a80 con 0x7ffa4c0af970 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c0af970 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994394529 0 0) 0x7ffa4c1d7200 con 0x7ffa4c09ac40 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c0b1a80 con 0x7ffa4c09ac40 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349364186 0 0) 0x7ffa4c0acd70 con 0x7ffa4c139360 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:32:11.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c1d7200 con 0x7ffa4c139360 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1260413264 0 0) 0x7ffa30004eb0 con 0x7ffa4c0af970 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c0acd70 con 0x7ffa4c0af970 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa48001bc0 con 0x7ffa4c0af970 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:11.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.804 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4757199 0 0) 0x7ffa4c0b1a80 con 0x7ffa4c09ac40 2024-03-29T11:32:11.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c09ac40 2024-03-29T11:32:11.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa3c001930 con 0x7ffa4c09ac40 2024-03-29T11:32:11.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:11.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:11.806 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.808 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 410343935 0 0) 0x7ffa4c1d7200 con 0x7ffa4c139360 2024-03-29T11:32:11.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c0b1a80 con 0x7ffa4c139360 2024-03-29T11:32:11.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa44001af0 con 0x7ffa4c139360 2024-03-29T11:32:11.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 599446109 0 0) 0x7ffa4c0acd70 con 0x7ffa4c0af970 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 >> v1:172.21.15.45:6790/0 conn(0x7ffa4c139360 legacy=0x7ffa4c1397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 >> v1:172.21.15.45:6789/0 conn(0x7ffa4c09ac40 legacy=0x7ffa4c0b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 monclient: found mon.b 2024-03-29T11:32:11.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0acf00 con 0x7ffa4c0af970 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.777+0000 7ffa41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 10 monclient: _renew_subs 2024-03-29T11:32:11.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c1d7da0 con 0x7ffa4c0af970 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa48004bf0 con 0x7ffa4c0af970 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa48004f00 con 0x7ffa4c0af970 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 10 monclient: dump: 2024-03-29T11:32:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.813 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 10 monclient: _renew_subs 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1e3b50 con 0x7ffa4c0af970 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa437fe640 4 set_mon_vals no callback set 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffa480053f0 con 0x7ffa4c0af970 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa53425640 1 librados: init done 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2274569042 0 0) 0x7ffa48005a50 con 0x7ffa4c0af970 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 10 client.4114.objecter ms_dispatch 0x7ffa4c000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:32:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.778+0000 7ffa41ffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.779+0000 7ffa53425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.779+0000 7ffa53425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.779+0000 7ffa53425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.779+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c11f9b0 con 0x7ffa4c0af970 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.781+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffa48005c10 con 0x7ffa4c0af970 2024-03-29T11:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.781+0000 7ffa41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.781+0000 7ffa41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:11.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.852+0000 7ffa53425640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4dc754de-2886-4a2f-8cd4-5c2501bfd966", "id": 1}] 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.852+0000 7ffa53425640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4dc754de-2886-4a2f-8cd4-5c2501bfd966", "id": 1}] 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.852+0000 7ffa53425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.852+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4dc754de-2886-4a2f-8cd4-5c2501bfd966", "id": 1} v 0) v1 -- 0x7ffa4c09b020 con 0x7ffa4c0af970 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.858+0000 7ffa41ffb640 1 -- 172.21.15.190:0/1878078390 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4dc754de-2886-4a2f-8cd4-5c2501bfd966", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 566652232 0 940483276) 0x7ffa48005db0 con 0x7ffa4c0af970 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.858+0000 7ffa41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4dc754de-2886-4a2f-8cd4-5c2501bfd966", "id": 1}] 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.858+0000 7ffa41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:11.859 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 10 librados: watch_flush enter 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 10 librados: watch_flush exit 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 10 monclient: shutdown 2024-03-29T11:32:11.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 >> v1:172.21.15.190:6789/0 conn(0x7ffa4c0af970 legacy=0x7ffa4c1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 shutdown_connections 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 >> 172.21.15.190:0/1878078390 conn(0x7ffa4c066b00 msgr2=0x7ffa4c13b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 shutdown_connections 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 -- 172.21.15.190:0/1878078390 wait complete. 2024-03-29T11:32:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.859+0000 7ffa53425640 1 librados: shutdown 2024-03-29T11:32:11.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new d3c56240-80be-4874-b7a7-1895d9be2c40 2 2024-03-29T11:32:11.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.977+0000 7f0fda286640 10 monclient: get_monmap_and_config 2024-03-29T11:32:11.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.977+0000 7f0fda286640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.977+0000 7f0fda286640 10 monclient: monmap: 2024-03-29T11:32:11.973 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:11.978860+0000 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:11.978860+0000 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.974 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.978+0000 7f0fda286640 1 Processor -- start 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.978+0000 7f0fda286640 1 -- start start 2024-03-29T11:32:11.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.978+0000 7f0fda286640 10 monclient: init 2024-03-29T11:32:11.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:11.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-b con 0x7f0fd409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-c con 0x7f0fd409c870 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-a con 0x7f0fd40b4360 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd409dec0 con 0x7f0fd40b4360 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd409da30 con 0x7f0fd409c870 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40b39c0 con 0x7f0fd409ac40 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient(hunting): authenticate will time out at 777.292303s 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd37fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0fd409c870 0x7f0fd413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35718/0 (socket says 172.21.15.190:35718) 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd37fe640 1 -- 172.21.15.190:0/1373683871 learned_addr learned my addr 172.21.15.190:0/1373683871 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52325961 0 0) 0x7f0fd409da30 con 0x7f0fd409c870 2024-03-29T11:32:11.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fbc003520 con 0x7f0fd409c870 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783836172 0 0) 0x7f0fcc001490 con 0x7f0fd40b4360 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd409da30 con 0x7f0fd40b4360 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982149980 0 0) 0x7f0fd40b39c0 con 0x7f0fd409ac40 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fcc001490 con 0x7f0fd409ac40 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 125385292 0 0) 0x7f0fbc003520 con 0x7f0fd409c870 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 >> v1:172.21.15.45:6790/0 conn(0x7f0fd409ac40 legacy=0x7f0fd40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 >> v1:172.21.15.45:6789/0 conn(0x7f0fd40b4360 legacy=0x7f0fd414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40b5160 con 0x7f0fd409c870 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fc4001690 con 0x7f0fd409c870 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:11.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: dump: 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.995 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fc4001870 con 0x7f0fd409c870 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fda286640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 4 set_mon_vals no callback set 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.979+0000 7f0fd2ffd640 1 -- 172.21.15.190:0/1373683871 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fc4001c80 con 0x7f0fd409c870 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fd2ffd640 10 monclient: dump: 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:11.996 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 10 monclient: shutdown 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 1 -- 172.21.15.190:0/1373683871 >> v1:172.21.15.190:6789/0 conn(0x7f0fd409c870 legacy=0x7f0fd413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 1 -- 172.21.15.190:0/1373683871 shutdown_connections 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 1 -- 172.21.15.190:0/1373683871 >> 172.21.15.190:0/1373683871 conn(0x7f0fd4066b00 msgr2=0x7f0fd4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 1 -- 172.21.15.190:0/1373683871 shutdown_connections 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.980+0000 7f0fda286640 1 -- 172.21.15.190:0/1373683871 wait complete. 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 10 monclient: monmap: 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:11.998 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 librados: starting msgr at 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 librados: starting objecter 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 Processor -- start 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 -- start start 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 librados: setting wanted keys 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 1 librados: calling monclient init 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.981+0000 7f0fda286640 10 monclient: init 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-c con 0x7f0fd409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-b con 0x7f0fd40b6b00 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient(hunting): picked mon.noname-a con 0x7f0fd41406b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd41d7e20 con 0x7f0fd41406b0 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd4140b00 con 0x7f0fd40b6b00 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40ad080 con 0x7f0fd409ac40 2024-03-29T11:32:11.999 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fda286640 10 monclient(hunting): authenticate will time out at 777.295337s 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd37fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0fd40b6b00 0x7f0fd413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35728/0 (socket says 172.21.15.190:35728) 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd37fe640 1 -- 172.21.15.190:0/923127668 learned_addr learned my addr 172.21.15.190:0/923127668 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f0fd40b6b00 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f0fd409ac40 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f0fd41406b0 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719652692 0 0) 0x7f0fd4140b00 con 0x7f0fd40b6b00 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:32:12.000 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc0004eb0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211078563 0 0) 0x7f0fd40ad080 con 0x7f0fd409ac40 2024-03-29T11:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4140b00 con 0x7f0fd409ac40 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796297169 0 0) 0x7f0fd41d7e20 con 0x7f0fd41406b0 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd40ad080 con 0x7f0fd41406b0 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2111608530 0 0) 0x7f0fc0004eb0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fd41d7e20 con 0x7f0fd40b6b00 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fc4001bc0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.003 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216076544 0 0) 0x7f0fd40ad080 con 0x7f0fd41406b0 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fc0004eb0 con 0x7f0fd41406b0 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fcc001a00 con 0x7f0fd41406b0 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.982+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542887212 0 0) 0x7f0fd4140b00 con 0x7f0fd409ac40 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fd40ad080 con 0x7f0fd409ac40 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fc8001820 con 0x7f0fd409ac40 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.005 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 939582672 0 0) 0x7f0fd41d7e20 con 0x7f0fd40b6b00 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 >> v1:172.21.15.45:6790/0 conn(0x7f0fd409ac40 legacy=0x7f0fd41d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 >> v1:172.21.15.45:6789/0 conn(0x7f0fd41406b0 legacy=0x7f0fd41e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 monclient: found mon.b 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.006 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd41d7ab0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 monclient: _renew_subs 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd41e3570 con 0x7f0fd40b6b00 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 monclient: _renew_subs 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd41e38e0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 1 librados: init done 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fc4004bf0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.007 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0fc4001230 con 0x7f0fd40b6b00 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd27fc640 4 set_mon_vals no callback set 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 monclient: dump: 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.008 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0fc40052d0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 756684433 0 0) 0x7f0fc4006920 con 0x7f0fd40b6b00 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 10 client.4120.objecter ms_dispatch 0x7f0fd4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fd0ff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.983+0000 7f0fda286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.984+0000 7f0fda286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.984+0000 7f0fda286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.984+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb4005080 con 0x7f0fd40b6b00 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.985+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0fc40065b0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.985+0000 7f0fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:11.985+0000 7f0fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:12.010 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:11.989+0000 7f2bbb020040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:12.010 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:11.997+0000 7fe430cc7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:32:12.040 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.045+0000 7f2bbb020040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:12.048 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.053+0000 7fe430cc7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:32:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.058+0000 7f0fda286640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d3c56240-80be-4874-b7a7-1895d9be2c40", "id": 2}] 2024-03-29T11:32:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.058+0000 7f0fda286640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d3c56240-80be-4874-b7a7-1895d9be2c40", "id": 2}] 2024-03-29T11:32:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.058+0000 7f0fda286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.058+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d3c56240-80be-4874-b7a7-1895d9be2c40", "id": 2} v 0) v1 -- 0x7f0fb40058b0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.065+0000 7f0fd0ff9640 1 -- 172.21.15.190:0/923127668 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3c56240-80be-4874-b7a7-1895d9be2c40", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3288344551 0 216599125) 0x7f0fc40057b0 con 0x7f0fd40b6b00 2024-03-29T11:32:12.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.065+0000 7f0fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d3c56240-80be-4874-b7a7-1895d9be2c40", "id": 2}] 2024-03-29T11:32:12.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.065+0000 7f0fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:12.061 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-03-29T11:32:12.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 10 librados: watch_flush enter 2024-03-29T11:32:12.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 10 librados: watch_flush exit 2024-03-29T11:32:12.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 10 monclient: shutdown 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 >> v1:172.21.15.190:6789/0 conn(0x7f0fd40b6b00 legacy=0x7f0fd413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 shutdown_connections 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 >> 172.21.15.190:0/923127668 conn(0x7f0fd4066b00 msgr2=0x7f0fd40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 shutdown_connections 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.066+0000 7f0fda286640 1 -- 172.21.15.190:0/923127668 wait complete. 2024-03-29T11:32:12.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.067+0000 7f0fda286640 1 librados: shutdown 2024-03-29T11:32:12.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new b1c06dd1-de39-4519-a58d-82501385cd63 3 2024-03-29T11:32:12.088 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.093+0000 7f2bbb020040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:32:12.095 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.100+0000 7fe430cc7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:32:12.145 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.150+0000 7f2bbb020040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:12.150 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.155+0000 7fe430cc7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.177+0000 7f616e663640 10 monclient: get_monmap_and_config 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.177+0000 7f616e663640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.177+0000 7f616e663640 10 monclient: monmap: 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:12.178203+0000 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:12.178203+0000 2024-03-29T11:32:12.172 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.173 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.173 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.173 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:12.173 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:12.173 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 1 Processor -- start 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 1 -- start start 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient: init 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient(hunting): picked mon.noname-a con 0x7f616809ac40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.191 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient(hunting): picked mon.noname-b con 0x7f616809c870 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient(hunting): picked mon.noname-c con 0x7f61680b4360 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616809dec0 con 0x7f616809ac40 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616809da30 con 0x7f61680b4360 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61680b39c0 con 0x7f616809c870 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616e663640 10 monclient(hunting): authenticate will time out at 777.491832s 2024-03-29T11:32:12.192 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616cc00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f61680b4360 0x7f616814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35730/0 (socket says 172.21.15.190:35730) 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.178+0000 7f616cc00640 1 -- 172.21.15.190:0/2591295549 learned_addr learned my addr 172.21.15.190:0/2591295549 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246369796 0 0) 0x7f616809da30 con 0x7f61680b4360 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6150003520 con 0x7f61680b4360 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953075365 0 0) 0x7f6158001110 con 0x7f616809ac40 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616809da30 con 0x7f616809ac40 2024-03-29T11:32:12.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388692603 0 0) 0x7f61680b39c0 con 0x7f616809c870 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158001110 con 0x7f616809c870 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1662225427 0 0) 0x7f6150003520 con 0x7f61680b4360 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 >> v1:172.21.15.45:6790/0 conn(0x7f616809c870 legacy=0x7f616813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 >> v1:172.21.15.45:6789/0 conn(0x7f616809ac40 legacy=0x7f61680b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 monclient: found mon.noname-c 2024-03-29T11:32:12.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61680b5160 con 0x7f61680b4360 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6160001a10 con 0x7f61680b4360 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:12.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: dump: 2024-03-29T11:32:12.196 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f616e663640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:32:12.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f616e663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6160001bf0 con 0x7f61680b4360 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 1 -- 172.21.15.190:0/2591295549 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6160002000 con 0x7f61680b4360 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f6166ffd640 10 monclient: dump: 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.199 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.200 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.200 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.200 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.201 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.201 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.201 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f616e663640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.179+0000 7f616e663640 4 set_mon_vals no callback set 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 10 monclient: shutdown 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 1 -- 172.21.15.190:0/2591295549 >> v1:172.21.15.190:6789/0 conn(0x7f61680b4360 legacy=0x7f616814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 1 -- 172.21.15.190:0/2591295549 shutdown_connections 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 1 -- 172.21.15.190:0/2591295549 >> 172.21.15.190:0/2591295549 conn(0x7f6168066b00 msgr2=0x7f6168066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 1 -- 172.21.15.190:0/2591295549 shutdown_connections 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.180+0000 7f616e663640 1 -- 172.21.15.190:0/2591295549 wait complete. 2024-03-29T11:32:12.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.181+0000 7f616e663640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.181+0000 7f616e663640 10 monclient: monmap: 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.181+0000 7f616e663640 1 librados: starting msgr at 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.181+0000 7f616e663640 1 librados: starting objecter 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.182+0000 7f616e663640 1 Processor -- start 2024-03-29T11:32:12.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.182+0000 7f616e663640 1 -- start start 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.182+0000 7f616e663640 1 librados: setting wanted keys 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.182+0000 7f616e663640 1 librados: calling monclient init 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.182+0000 7f616e663640 10 monclient: init 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient(hunting): picked mon.noname-b con 0x7f616809ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient(hunting): picked mon.noname-c con 0x7f6168141910 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient(hunting): picked mon.noname-a con 0x7f616813ffa0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6168139b60 con 0x7f616813ffa0 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61681d6f10 con 0x7f616809ac40 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61680afba0 con 0x7f6168141910 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient(hunting): authenticate will time out at 777.496252s 2024-03-29T11:32:12.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6167fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f616809ac40 0x7f6168141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35734/0 (socket says 172.21.15.190:35734) 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6167fff640 1 -- 172.21.15.190:0/4161575374 learned_addr learned my addr 172.21.15.190:0/4161575374 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter ms_handle_connect 0x7f616809ac40 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter ms_handle_connect 0x7f6168141910 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter ms_handle_connect 0x7f616813ffa0 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400687193 0 0) 0x7f61681d6f10 con 0x7f616809ac40 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154004eb0 con 0x7f616809ac40 2024-03-29T11:32:12.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021825218 0 0) 0x7f61680afba0 con 0x7f6168141910 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61681d6f10 con 0x7f6168141910 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884644641 0 0) 0x7f61680b5160 con 0x7f616813ffa0 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61680afba0 con 0x7f616813ffa0 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165270602 0 0) 0x7f6154004eb0 con 0x7f616809ac40 2024-03-29T11:32:12.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61680b5160 con 0x7f616809ac40 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f61580019a0 con 0x7f616809ac40 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4204839258 0 0) 0x7f61681d6f10 con 0x7f6168141910 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6154004eb0 con 0x7f6168141910 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389272876 0 0) 0x7f61680afba0 con 0x7f616813ffa0 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61681d6f10 con 0x7f616813ffa0 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f615c0018b0 con 0x7f6168141910 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6160001d60 con 0x7f616813ffa0 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4213026110 0 0) 0x7f61680b5160 con 0x7f616809ac40 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 >> v1:172.21.15.45:6790/0 conn(0x7f6168141910 legacy=0x7f61681d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 >> v1:172.21.15.45:6789/0 conn(0x7f616813ffa0 legacy=0x7f6168138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 monclient: found mon.b 2024-03-29T11:32:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6168139520 con 0x7f616809ac40 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f6164ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient: _renew_subs 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.183+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6168139710 con 0x7f616809ac40 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:12.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 monclient: _renew_subs 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61681d7c90 con 0x7f616809ac40 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 1 librados: init done 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6158001c40 con 0x7f616809ac40 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6158003a60 con 0x7f616809ac40 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f61667fc640 4 set_mon_vals no callback set 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 10 monclient: dump: 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6158003fd0 con 0x7f616809ac40 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 531023503 0 0) 0x7f6158006780 con 0x7f616809ac40 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 10 client.4126.objecter ms_dispatch 0x7f6168000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:32:12.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f6164ff9640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.184+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148005080 con 0x7f616809ac40 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.188+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f61580043c0 con 0x7f616809ac40 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.188+0000 7f6164ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.188+0000 7f6164ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:12.215 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.198+0000 7f2bbb020040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:12.215 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.200+0000 7fe430cc7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:32:12.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.258+0000 7f616e663640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b1c06dd1-de39-4519-a58d-82501385cd63", "id": 3}] 2024-03-29T11:32:12.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.259+0000 7f616e663640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b1c06dd1-de39-4519-a58d-82501385cd63", "id": 3}] 2024-03-29T11:32:12.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.259+0000 7f616e663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.259+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b1c06dd1-de39-4519-a58d-82501385cd63", "id": 3} v 0) v1 -- 0x7f61480058b0 con 0x7f616809ac40 2024-03-29T11:32:12.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.266+0000 7f6164ff9640 1 -- 172.21.15.190:0/4161575374 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1c06dd1-de39-4519-a58d-82501385cd63", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 4283582662 0 525046306) 0x7f6158006480 con 0x7f616809ac40 2024-03-29T11:32:12.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.266+0000 7f6164ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b1c06dd1-de39-4519-a58d-82501385cd63", "id": 3}] 2024-03-29T11:32:12.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.266+0000 7f6164ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:12.262 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-03-29T11:32:12.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 10 librados: watch_flush enter 2024-03-29T11:32:12.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 10 librados: watch_flush exit 2024-03-29T11:32:12.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:32:12.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 10 monclient: shutdown 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 >> v1:172.21.15.190:6789/0 conn(0x7f616809ac40 legacy=0x7f6168141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 shutdown_connections 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.267+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 >> 172.21.15.190:0/4161575374 conn(0x7f6168066b00 msgr2=0x7f616813c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.268+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 shutdown_connections 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.268+0000 7f616e663640 1 -- 172.21.15.190:0/4161575374 wait complete. 2024-03-29T11:32:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.268+0000 7f616e663640 1 librados: shutdown 2024-03-29T11:32:12.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new aa6ed173-15f2-407a-823d-f5da2f886d5e 4 2024-03-29T11:32:12.313 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.318+0000 7f2bbb020040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:12.315 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.320+0000 7fe430cc7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:32:12.387 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.392+0000 7f2bbb020040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:12.387 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.392+0000 7fe430cc7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.417+0000 7fe3abc14640 10 monclient: get_monmap_and_config 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.417+0000 7fe3abc14640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.417+0000 7fe3abc14640 10 monclient: monmap: 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:12.418874+0000 2024-03-29T11:32:12.413 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:12.418874+0000 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.418+0000 7fe3abc14640 1 Processor -- start 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.418+0000 7fe3abc14640 1 -- start start 2024-03-29T11:32:12.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.418+0000 7fe3abc14640 10 monclient: init 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a409ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a409c870 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a40b4360 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a409dec0 con 0x7fe3a409c870 2024-03-29T11:32:12.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a409da30 con 0x7fe3a409ac40 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40b39c0 con 0x7fe3a40b4360 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 10 monclient(hunting): authenticate will time out at 777.732332s 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a99b0640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe3a409ac40 0x7fe3a40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35736/0 (socket says 172.21.15.190:35736) 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a99b0640 1 -- 172.21.15.190:0/1795527727 learned_addr learned my addr 172.21.15.190:0/1795527727 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643059637 0 0) 0x7fe3a409da30 con 0x7fe3a409ac40 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe390003520 con 0x7fe3a409ac40 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564128250 0 0) 0x7fe3a0001110 con 0x7fe3a409c870 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a409da30 con 0x7fe3a409c870 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593262142 0 0) 0x7fe3a40b39c0 con 0x7fe3a40b4360 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a0001110 con 0x7fe3a40b4360 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916385864 0 0) 0x7fe390003520 con 0x7fe3a409ac40 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 >> v1:172.21.15.45:6790/0 conn(0x7fe3a40b4360 legacy=0x7fe3a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 >> v1:172.21.15.45:6789/0 conn(0x7fe3a409c870 legacy=0x7fe3a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 monclient: found mon.noname-c 2024-03-29T11:32:12.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40b5160 con 0x7fe3a409ac40 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe394001690 con 0x7fe3a409ac40 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.435 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3a89ae640 10 monclient: dump: 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.436 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.419+0000 7fe3abc14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3940017f0 con 0x7fe3a409ac40 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 1 -- 172.21.15.190:0/1795527727 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe394001c00 con 0x7fe3a409ac40 2024-03-29T11:32:12.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 4 set_mon_vals no callback set 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3a89ae640 10 monclient: dump: 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.438 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 10 monclient: shutdown 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 1 -- 172.21.15.190:0/1795527727 >> v1:172.21.15.190:6789/0 conn(0x7fe3a409ac40 legacy=0x7fe3a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 1 -- 172.21.15.190:0/1795527727 shutdown_connections 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 1 -- 172.21.15.190:0/1795527727 >> 172.21.15.190:0/1795527727 conn(0x7fe3a4066b00 msgr2=0x7fe3a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 1 -- 172.21.15.190:0/1795527727 shutdown_connections 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.420+0000 7fe3abc14640 1 -- 172.21.15.190:0/1795527727 wait complete. 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 10 monclient: monmap: 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 librados: starting msgr at 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 librados: starting objecter 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 Processor -- start 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 -- start start 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 librados: setting wanted keys 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 1 librados: calling monclient init 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.421+0000 7fe3abc14640 10 monclient: init 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a409ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a41c9870 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a40b0f60 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a41c9e20 con 0x7fe3a40b0f60 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a413ffa0 con 0x7fe3a41c9870 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a4141ef0 con 0x7fe3a409ac40 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient(hunting): authenticate will time out at 777.735235s 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3a91af640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe3a41c9870 0x7fe3a4139d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35746/0 (socket says 172.21.15.190:35746) 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3a91af640 1 -- 172.21.15.190:0/2965059467 learned_addr learned my addr 172.21.15.190:0/2965059467 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter ms_handle_connect 0x7fe3a41c9870 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter ms_handle_connect 0x7fe3a40b0f60 2024-03-29T11:32:12.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter ms_handle_connect 0x7fe3a409ac40 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698102194 0 0) 0x7fe3a413ffa0 con 0x7fe3a41c9870 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe388004eb0 con 0x7fe3a41c9870 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76193419 0 0) 0x7fe3a41c9e20 con 0x7fe3a40b0f60 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a413ffa0 con 0x7fe3a40b0f60 2024-03-29T11:32:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806977984 0 0) 0x7fe3a4141ef0 con 0x7fe3a409ac40 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a41c9e20 con 0x7fe3a409ac40 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645877902 0 0) 0x7fe388004eb0 con 0x7fe3a41c9870 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3a4141ef0 con 0x7fe3a41c9870 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe3a0001a20 con 0x7fe3a41c9870 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 121834691 0 0) 0x7fe3a413ffa0 con 0x7fe3a40b0f60 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe388004eb0 con 0x7fe3a40b0f60 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe39c001b50 con 0x7fe3a40b0f60 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:12.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581277634 0 0) 0x7fe3a41c9e20 con 0x7fe3a409ac40 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3a413ffa0 con 0x7fe3a409ac40 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe394001960 con 0x7fe3a409ac40 2024-03-29T11:32:12.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2027963822 0 0) 0x7fe3a4141ef0 con 0x7fe3a41c9870 2024-03-29T11:32:12.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 >> v1:172.21.15.45:6790/0 conn(0x7fe3a409ac40 legacy=0x7fe3a41425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 >> v1:172.21.15.45:6789/0 conn(0x7fe3a40b0f60 legacy=0x7fe3a41e2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 monclient: found mon.b 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40b77c0 con 0x7fe3a41c9870 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe39a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _renew_subs 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a40b7a10 con 0x7fe3a41c9870 2024-03-29T11:32:12.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _renew_subs 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a4142e90 con 0x7fe3a41c9870 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.422+0000 7fe3abc14640 1 librados: init done 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3a0001ce0 con 0x7fe3a41c9870 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe3a0003a60 con 0x7fe3a41c9870 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39bfff640 4 set_mon_vals no callback set 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 10 monclient: dump: 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.449 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe3a0003f50 con 0x7fe3a41c9870 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4145817750 0 0) 0x7fe3a00068a0 con 0x7fe3a41c9870 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 10 client.4132.objecter ms_dispatch 0x7fe3a4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe39a7fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:32:12.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe3abc14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe3abc14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe3abc14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.423+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c005080 con 0x7fe3a41c9870 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.427+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe3a0006b80 con 0x7fe3a41c9870 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.427+0000 7fe39a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.427+0000 7fe39a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:12.466 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.443+0000 7f2bbb020040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:12.466 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.440+0000 7fe430cc7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:32:12.490 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.495+0000 7fe430cc7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:32:12.491 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.497+0000 7f2bbb020040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:32:12.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.499+0000 7fe3abc14640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aa6ed173-15f2-407a-823d-f5da2f886d5e", "id": 4}] 2024-03-29T11:32:12.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.499+0000 7fe3abc14640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aa6ed173-15f2-407a-823d-f5da2f886d5e", "id": 4}] 2024-03-29T11:32:12.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.499+0000 7fe3abc14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.499+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aa6ed173-15f2-407a-823d-f5da2f886d5e", "id": 4} v 0) v1 -- 0x7fe38c0058b0 con 0x7fe3a41c9870 2024-03-29T11:32:12.498 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe39a7fc640 1 -- 172.21.15.190:0/2965059467 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aa6ed173-15f2-407a-823d-f5da2f886d5e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4233453154 0 1697012583) 0x7fe3a0006390 con 0x7fe3a41c9870 2024-03-29T11:32:12.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe39a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aa6ed173-15f2-407a-823d-f5da2f886d5e", "id": 4}] 2024-03-29T11:32:12.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe39a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:12.499 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-03-29T11:32:12.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 10 librados: watch_flush enter 2024-03-29T11:32:12.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 10 librados: watch_flush exit 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 10 monclient: shutdown 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.504+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 >> v1:172.21.15.190:6789/0 conn(0x7fe3a41c9870 legacy=0x7fe3a4139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.505+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 shutdown_connections 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.505+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 >> 172.21.15.190:0/2965059467 conn(0x7fe3a4066b00 msgr2=0x7fe3a40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.505+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 shutdown_connections 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.505+0000 7fe3abc14640 1 -- 172.21.15.190:0/2965059467 wait complete. 2024-03-29T11:32:12.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.505+0000 7fe3abc14640 1 librados: shutdown 2024-03-29T11:32:12.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 86551d84-ea1f-4a90-b4d0-62b7dbff72b7 5 2024-03-29T11:32:12.539 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.544+0000 7fe430cc7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:32:12.543 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.549+0000 7f2bbb020040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:32:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.615+0000 7ff88bb79640 10 monclient: get_monmap_and_config 2024-03-29T11:32:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.615+0000 7ff88bb79640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.615+0000 7ff88bb79640 10 monclient: monmap: 2024-03-29T11:32:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:12.616867+0000 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:12.616867+0000 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.616+0000 7ff88bb79640 1 Processor -- start 2024-03-29T11:32:12.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.616+0000 7ff88bb79640 1 -- start start 2024-03-29T11:32:12.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.616+0000 7ff88bb79640 10 monclient: init 2024-03-29T11:32:12.626 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.626 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:12.626 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-c con 0x7ff88409ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.626 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-a con 0x7ff88409c870 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.626 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-b con 0x7ff8840b4360 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88409dec0 con 0x7ff88409c870 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88409da30 con 0x7ff88409ac40 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840b39c0 con 0x7ff8840b4360 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff88bb79640 10 monclient(hunting): authenticate will time out at 777.930445s 2024-03-29T11:32:12.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff889915640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff88409ac40 0x7ff8840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35748/0 (socket says 172.21.15.190:35748) 2024-03-29T11:32:12.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff889915640 1 -- 172.21.15.190:0/197127958 learned_addr learned my addr 172.21.15.190:0/197127958 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529106349 0 0) 0x7ff88409da30 con 0x7ff88409ac40 2024-03-29T11:32:12.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003520 con 0x7ff88409ac40 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202909942 0 0) 0x7ff8840b39c0 con 0x7ff8840b4360 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88409da30 con 0x7ff8840b4360 2024-03-29T11:32:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20633198 0 0) 0x7ff880001110 con 0x7ff88409c870 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8840b39c0 con 0x7ff88409c870 2024-03-29T11:32:12.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414119802 0 0) 0x7ff870003520 con 0x7ff88409ac40 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 >> v1:172.21.15.45:6790/0 conn(0x7ff8840b4360 legacy=0x7ff88414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 >> v1:172.21.15.45:6789/0 conn(0x7ff88409c870 legacy=0x7ff88413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 monclient: found mon.noname-c 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8840b5160 con 0x7ff88409ac40 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.617+0000 7ff888913640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff874001690 con 0x7ff88409ac40 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:12.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: dump: 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.637 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff874001870 con 0x7ff88409ac40 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 1 -- 172.21.15.190:0/197127958 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff874001c80 con 0x7ff88409ac40 2024-03-29T11:32:12.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 4 set_mon_vals no callback set 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff888913640 10 monclient: dump: 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.639 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 10 monclient: shutdown 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 1 -- 172.21.15.190:0/197127958 >> v1:172.21.15.190:6789/0 conn(0x7ff88409ac40 legacy=0x7ff8840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 1 -- 172.21.15.190:0/197127958 shutdown_connections 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 1 -- 172.21.15.190:0/197127958 >> 172.21.15.190:0/197127958 conn(0x7ff884066b00 msgr2=0x7ff884066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 1 -- 172.21.15.190:0/197127958 shutdown_connections 2024-03-29T11:32:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.618+0000 7ff88bb79640 1 -- 172.21.15.190:0/197127958 wait complete. 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 10 monclient: monmap: 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:12.641 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 librados: starting msgr at 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 librados: starting objecter 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 Processor -- start 2024-03-29T11:32:12.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 -- start start 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 librados: setting wanted keys 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 1 librados: calling monclient init 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.619+0000 7ff88bb79640 10 monclient: init 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-a con 0x7ff88409ac40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-b con 0x7ff8840ad140 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient(hunting): picked mon.noname-c con 0x7ff8841d77e0 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8841d7c20 con 0x7ff88409ac40 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8841d7db0 con 0x7ff8840ad140 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8841d7f40 con 0x7ff8841d77e0 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff88bb79640 10 monclient(hunting): authenticate will time out at 777.933313s 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff889114640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff8840ad140 0x7ff8840afc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35752/0 (socket says 172.21.15.190:35752) 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff889114640 1 -- 172.21.15.190:0/993498212 learned_addr learned my addr 172.21.15.190:0/993498212 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter ms_handle_connect 0x7ff8840ad140 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter ms_handle_connect 0x7ff88409ac40 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter ms_handle_connect 0x7ff8841d77e0 2024-03-29T11:32:12.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775996788 0 0) 0x7ff8841d7db0 con 0x7ff8840ad140 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff868004eb0 con 0x7ff8840ad140 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457107982 0 0) 0x7ff8841d7c20 con 0x7ff88409ac40 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8841d7db0 con 0x7ff88409ac40 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986908685 0 0) 0x7ff8841d7f40 con 0x7ff8841d77e0 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8841d7c20 con 0x7ff8841d77e0 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593440043 0 0) 0x7ff868004eb0 con 0x7ff8840ad140 2024-03-29T11:32:12.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8841d7f40 con 0x7ff8840ad140 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff8800017e0 con 0x7ff8840ad140 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3558363801 0 0) 0x7ff8841d7db0 con 0x7ff88409ac40 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff868004eb0 con 0x7ff88409ac40 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff874001b80 con 0x7ff88409ac40 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.649 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1600030970 0 0) 0x7ff8841d7c20 con 0x7ff8841d77e0 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.620+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8841d7db0 con 0x7ff8841d77e0 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff87c001b50 con 0x7ff8841d77e0 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient(hunting): dump: 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.650 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1679790453 0 0) 0x7ff8841d7f40 con 0x7ff8840ad140 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 >> v1:172.21.15.45:6790/0 conn(0x7ff8841d77e0 legacy=0x7ff8841e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 >> v1:172.21.15.45:6789/0 conn(0x7ff88409ac40 legacy=0x7ff8840b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 monclient: found mon.b 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8841e6390 con 0x7ff8840ad140 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 10 monclient: _renew_subs 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8841e3360 con 0x7ff8840ad140 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 10 monclient: _renew_subs 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8841e38e0 con 0x7ff8840ad140 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff88bb79640 1 librados: init done 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff880001d50 con 0x7ff8840ad140 2024-03-29T11:32:12.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff8800037b0 con 0x7ff8840ad140 2024-03-29T11:32:12.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87bfff640 4 set_mon_vals no callback set 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 monclient: dump: 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff880003d20 con 0x7ff8840ad140 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1310911635 0 0) 0x7ff880006630 con 0x7ff8840ad140 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 10 client.4138.objecter ms_dispatch 0x7ff884000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:32:12.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.621+0000 7ff87a7fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.622+0000 7ff88bb79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.622+0000 7ff88bb79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.622+0000 7ff88bb79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.622+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c005080 con 0x7ff8840ad140 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.625+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff880004250 con 0x7ff8840ad140 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.625+0000 7ff87a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.625+0000 7ff87a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:12.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.697+0000 7ff88bb79640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "86551d84-ea1f-4a90-b4d0-62b7dbff72b7", "id": 5}] 2024-03-29T11:32:12.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.697+0000 7ff88bb79640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "86551d84-ea1f-4a90-b4d0-62b7dbff72b7", "id": 5}] 2024-03-29T11:32:12.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.697+0000 7ff88bb79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.697+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "86551d84-ea1f-4a90-b4d0-62b7dbff72b7", "id": 5} v 0) v1 -- 0x7ff86c0058b0 con 0x7ff8840ad140 2024-03-29T11:32:12.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.702+0000 7ff87a7fc640 1 -- 172.21.15.190:0/993498212 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "86551d84-ea1f-4a90-b4d0-62b7dbff72b7", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2369915740 0 1988412176) 0x7ff8800057c0 con 0x7ff8840ad140 2024-03-29T11:32:12.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.702+0000 7ff87a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "86551d84-ea1f-4a90-b4d0-62b7dbff72b7", "id": 5}] 2024-03-29T11:32:12.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.702+0000 7ff87a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:12.697 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-03-29T11:32:12.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 10 librados: watch_flush enter 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 10 librados: watch_flush exit 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 10 monclient: shutdown 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 >> v1:172.21.15.190:6789/0 conn(0x7ff8840ad140 legacy=0x7ff8840afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.699 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 shutdown_connections 2024-03-29T11:32:12.700 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 >> 172.21.15.190:0/993498212 conn(0x7ff884066b00 msgr2=0x7ff88413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.700 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 shutdown_connections 2024-03-29T11:32:12.700 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 -- 172.21.15.190:0/993498212 wait complete. 2024-03-29T11:32:12.700 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.703+0000 7ff88bb79640 1 librados: shutdown 2024-03-29T11:32:12.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 7766d34a-8266-4da6-9cef-bf66289c4ce5 6 2024-03-29T11:32:12.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.853+0000 7fd240ef1640 10 monclient: get_monmap_and_config 2024-03-29T11:32:12.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.853+0000 7fd240ef1640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.853+0000 7fd240ef1640 10 monclient: monmap: 2024-03-29T11:32:12.848 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:12.854769+0000 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:12.854769+0000 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:12.849 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.854+0000 7fd240ef1640 1 Processor -- start 2024-03-29T11:32:12.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.854+0000 7fd240ef1640 1 -- start start 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.854+0000 7fd240ef1640 10 monclient: init 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c09ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c09c870 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c0b4360 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c09dec0 con 0x7fd23c0b4360 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c09da30 con 0x7fd23c09ac40 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c0b39c0 con 0x7fd23c09c870 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient(hunting): authenticate will time out at 778.168256s 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd23a59c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd23c09ac40 0x7fd23c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35756/0 (socket says 172.21.15.190:35756) 2024-03-29T11:32:12.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd23a59c640 1 -- 172.21.15.190:0/1789587938 learned_addr learned my addr 172.21.15.190:0/1789587938 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595443334 0 0) 0x7fd23c09da30 con 0x7fd23c09ac40 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd228003520 con 0x7fd23c09ac40 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899229127 0 0) 0x7fd2340013c0 con 0x7fd23c0b4360 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c09da30 con 0x7fd23c0b4360 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067158032 0 0) 0x7fd23c0b39c0 con 0x7fd23c09c870 2024-03-29T11:32:12.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2340013c0 con 0x7fd23c09c870 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082620443 0 0) 0x7fd228003520 con 0x7fd23c09ac40 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 >> v1:172.21.15.45:6790/0 conn(0x7fd23c09c870 legacy=0x7fd23c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 >> v1:172.21.15.45:6789/0 conn(0x7fd23c0b4360 legacy=0x7fd23c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 monclient: found mon.noname-c 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c0b5160 con 0x7fd23c09ac40 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd22c001690 con 0x7fd23c09ac40 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: dump: 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.908 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd22c001870 con 0x7fd23c09ac40 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd240ef1640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.855+0000 7fd239d9b640 1 -- 172.21.15.190:0/1789587938 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd22c001c80 con 0x7fd23c09ac40 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 4 set_mon_vals no callback set 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd239d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd239d9b640 10 monclient: dump: 2024-03-29T11:32:12.918 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 10 monclient: shutdown 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 1 -- 172.21.15.190:0/1789587938 >> v1:172.21.15.190:6789/0 conn(0x7fd23c09ac40 legacy=0x7fd23c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 1 -- 172.21.15.190:0/1789587938 shutdown_connections 2024-03-29T11:32:12.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 1 -- 172.21.15.190:0/1789587938 >> 172.21.15.190:0/1789587938 conn(0x7fd23c066b00 msgr2=0x7fd23c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 1 -- 172.21.15.190:0/1789587938 shutdown_connections 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.856+0000 7fd240ef1640 1 -- 172.21.15.190:0/1789587938 wait complete. 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 10 monclient: build_initial_monmap 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 10 monclient: monmap: 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.920 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:12.921 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 librados: starting msgr at 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 librados: starting objecter 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 Processor -- start 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 -- start start 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 librados: setting wanted keys 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 1 librados: calling monclient init 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.857+0000 7fd240ef1640 10 monclient: init 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:12.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:12.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c09ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:12.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c0b12b0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:12.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c1d7a40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:12.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c0b8000 con 0x7fd23c1d7a40 2024-03-29T11:32:12.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c142160 con 0x7fd23c0b12b0 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c142f70 con 0x7fd23c09ac40 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd240ef1640 10 monclient(hunting): authenticate will time out at 778.171402s 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd233fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd23c0b12b0 0x7fd23c1d7330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35770/0 (socket says 172.21.15.190:35770) 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd233fff640 1 -- 172.21.15.190:0/2576116869 learned_addr learned my addr 172.21.15.190:0/2576116869 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter ms_handle_connect 0x7fd23c0b12b0 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter ms_handle_connect 0x7fd23c09ac40 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter ms_handle_connect 0x7fd23c1d7a40 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454293660 0 0) 0x7fd23c142160 con 0x7fd23c0b12b0 2024-03-29T11:32:12.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd21c004eb0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770254873 0 0) 0x7fd23c142f70 con 0x7fd23c09ac40 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c142160 con 0x7fd23c09ac40 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009681288 0 0) 0x7fd23c0b8000 con 0x7fd23c1d7a40 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:12.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c142f70 con 0x7fd23c1d7a40 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070270780 0 0) 0x7fd21c004eb0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd23c0b8000 con 0x7fd23c0b12b0 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd224001a10 con 0x7fd23c0b12b0 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.929 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206364809 0 0) 0x7fd23c142160 con 0x7fd23c09ac40 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd21c004eb0 con 0x7fd23c09ac40 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd22c001b80 con 0x7fd23c09ac40 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.858+0000 7fd232ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.930 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000561758 0 0) 0x7fd23c142f70 con 0x7fd23c1d7a40 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd23c142160 con 0x7fd23c1d7a40 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd234001970 con 0x7fd23c1d7a40 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.931 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2278293650 0 0) 0x7fd23c0b8000 con 0x7fd23c0b12b0 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 >> v1:172.21.15.45:6790/0 conn(0x7fd23c09ac40 legacy=0x7fd23c142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 >> v1:172.21.15.45:6789/0 conn(0x7fd23c1d7a40 legacy=0x7fd23c1e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 monclient: found mon.b 2024-03-29T11:32:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:12.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c1d7ed0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:12.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:12.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:12.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 10 monclient: _renew_subs 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c1e3400 con 0x7fd23c0b12b0 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 10 monclient: _renew_subs 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c1e3770 con 0x7fd23c0b12b0 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd240ef1640 1 librados: init done 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd224001cf0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd224003a60 con 0x7fd23c0b12b0 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:12.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 monclient: dump: 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:12.950 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd23959a640 4 set_mon_vals no callback set 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd224003fd0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3364474955 0 0) 0x7fd2240069d0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 10 client.4144.objecter ms_dispatch 0x7fd23c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:32:12.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.859+0000 7fd232ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.860+0000 7fd240ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.860+0000 7fd240ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.860+0000 7fd240ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.860+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd220005080 con 0x7fd23c0b12b0 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.862+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd2240063e0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.862+0000 7fd232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:12.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.862+0000 7fd232ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:12.953 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:12.941+0000 7fe430cc7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:12.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.934+0000 7fd240ef1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7766d34a-8266-4da6-9cef-bf66289c4ce5", "id": 6}] 2024-03-29T11:32:12.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.934+0000 7fd240ef1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7766d34a-8266-4da6-9cef-bf66289c4ce5", "id": 6}] 2024-03-29T11:32:12.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.934+0000 7fd240ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:12.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.934+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7766d34a-8266-4da6-9cef-bf66289c4ce5", "id": 6} v 0) v1 -- 0x7fd2200058b0 con 0x7fd23c0b12b0 2024-03-29T11:32:12.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.938+0000 7fd232ffd640 1 -- 172.21.15.190:0/2576116869 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7766d34a-8266-4da6-9cef-bf66289c4ce5", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1513069396 0 1113811849) 0x7fd224006580 con 0x7fd23c0b12b0 2024-03-29T11:32:12.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.938+0000 7fd232ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7766d34a-8266-4da6-9cef-bf66289c4ce5", "id": 6}] 2024-03-29T11:32:12.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.938+0000 7fd232ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:12.954 INFO:teuthology.orchestra.run.smithi190.stdout:6 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 10 librados: watch_flush enter 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 10 librados: watch_flush exit 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 10 monclient: shutdown 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 >> v1:172.21.15.190:6789/0 conn(0x7fd23c0b12b0 legacy=0x7fd23c1d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 shutdown_connections 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 >> 172.21.15.190:0/2576116869 conn(0x7fd23c066b00 msgr2=0x7fd23c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 shutdown_connections 2024-03-29T11:32:12.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 -- 172.21.15.190:0/2576116869 wait complete. 2024-03-29T11:32:12.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:12.939+0000 7fd240ef1640 1 librados: shutdown 2024-03-29T11:32:12.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new b3d19353-b793-4b49-9361-aaf623b141ea 7 2024-03-29T11:32:12.957 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:12.954+0000 7f2bbb020040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.061+0000 7fb687055640 10 monclient: get_monmap_and_config 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.061+0000 7fb687055640 10 monclient: build_initial_monmap 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.061+0000 7fb687055640 10 monclient: monmap: 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:32:13.062697+0000 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:32:13.062697+0000 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:13.057 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:13.058 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:13.058 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.062+0000 7fb687055640 1 Processor -- start 2024-03-29T11:32:13.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.062+0000 7fb687055640 1 -- start start 2024-03-29T11:32:13.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.062+0000 7fb687055640 10 monclient: init 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient(hunting): picked mon.noname-b con 0x7fb68009ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient(hunting): picked mon.noname-c con 0x7fb68009c870 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient(hunting): picked mon.noname-a con 0x7fb6800b4360 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68009dec0 con 0x7fb6800b4360 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68009da30 con 0x7fb68009c870 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800b39c0 con 0x7fb68009ac40 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:13.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient(hunting): authenticate will time out at 778.376120s 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67ffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb68009c870 0x7fb68013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35782/0 (socket says 172.21.15.190:35782) 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67ffff640 1 -- 172.21.15.190:0/2450953246 learned_addr learned my addr 172.21.15.190:0/2450953246 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701043356 0 0) 0x7fb68009da30 con 0x7fb68009c870 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb670003520 con 0x7fb68009c870 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415992089 0 0) 0x7fb6800b39c0 con 0x7fb68009ac40 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:32:13.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68009da30 con 0x7fb68009ac40 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416540457 0 0) 0x7fb678001490 con 0x7fb6800b4360 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6800b39c0 con 0x7fb6800b4360 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588320748 0 0) 0x7fb670003520 con 0x7fb68009c870 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 >> v1:172.21.15.45:6790/0 conn(0x7fb68009ac40 legacy=0x7fb6800b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 >> v1:172.21.15.45:6789/0 conn(0x7fb6800b4360 legacy=0x7fb68014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6800b5160 con 0x7fb68009c870 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb66c001690 con 0x7fb68009c870 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: dump: 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.073 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb66c001870 con 0x7fb68009c870 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 1 -- 172.21.15.190:0/2450953246 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb66c001c80 con 0x7fb68009c870 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb687055640 4 set_mon_vals no callback set 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:13.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.063+0000 7fb67f7fe640 10 monclient: dump: 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.075 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.098 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.098 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.098 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.098 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 10 monclient: shutdown 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 1 -- 172.21.15.190:0/2450953246 >> v1:172.21.15.190:6789/0 conn(0x7fb68009c870 legacy=0x7fb68013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 1 -- 172.21.15.190:0/2450953246 shutdown_connections 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 1 -- 172.21.15.190:0/2450953246 >> 172.21.15.190:0/2450953246 conn(0x7fb680066b00 msgr2=0x7fb680066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 1 -- 172.21.15.190:0/2450953246 shutdown_connections 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.064+0000 7fb687055640 1 -- 172.21.15.190:0/2450953246 wait complete. 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 10 monclient: build_initial_monmap 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 10 monclient: monmap: 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 librados: starting msgr at 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 librados: starting objecter 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 Processor -- start 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 -- start start 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 librados: setting wanted keys 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 1 librados: calling monclient init 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.065+0000 7fb687055640 10 monclient: init 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:13.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient(hunting): picked mon.noname-a con 0x7fb68009ac40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient(hunting): picked mon.noname-b con 0x7fb6800afa60 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient(hunting): picked mon.noname-c con 0x7fb6801db5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800af690 con 0x7fb68009ac40 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68011fbb0 con 0x7fb6800afa60 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800afef0 con 0x7fb6801db5a0 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb687055640 10 monclient(hunting): authenticate will time out at 778.379269s 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67ffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb6800afa60 0x7fb6800b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:35796/0 (socket says 172.21.15.190:35796) 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67ffff640 1 -- 172.21.15.190:0/1802931244 learned_addr learned my addr 172.21.15.190:0/1802931244 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter ms_handle_connect 0x7fb6800afa60 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter ms_handle_connect 0x7fb68009ac40 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter ms_handle_connect 0x7fb6801db5a0 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462287582 0 0) 0x7fb68011fbb0 con 0x7fb6800afa60 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664004eb0 con 0x7fb6800afa60 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499996333 0 0) 0x7fb6800af690 con 0x7fb68009ac40 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68011fbb0 con 0x7fb68009ac40 2024-03-29T11:32:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035759078 0 0) 0x7fb6800afef0 con 0x7fb6801db5a0 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6800af690 con 0x7fb6801db5a0 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878138807 0 0) 0x7fb664004eb0 con 0x7fb6800afa60 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6800afef0 con 0x7fb6800afa60 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb66c001bc0 con 0x7fb6800afa60 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): dump: 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.103 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319830176 0 0) 0x7fb6800af690 con 0x7fb6801db5a0 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb664004eb0 con 0x7fb6801db5a0 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb678001bc0 con 0x7fb6801db5a0 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 10 monclient(hunting): dump: 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.104 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2074799086 0 0) 0x7fb68011fbb0 con 0x7fb68009ac40 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6800af690 con 0x7fb68009ac40 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.066+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb6740018b0 con 0x7fb68009ac40 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient(hunting): dump: 2024-03-29T11:32:13.106 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.106 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.106 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.106 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.106 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.114 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.114 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.114 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2607511333 0 0) 0x7fb6800afef0 con 0x7fb6800afa60 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 >> v1:172.21.15.45:6790/0 conn(0x7fb6801db5a0 legacy=0x7fb6801e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 >> v1:172.21.15.45:6789/0 conn(0x7fb68009ac40 legacy=0x7fb680139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 monclient: found mon.b 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6801dab60 con 0x7fb6800afa60 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 monclient: _renew_subs 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6801dadd0 con 0x7fb6800afa60 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 monclient: _renew_subs 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb680139d80 con 0x7fb6800afa60 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 1 librados: init done 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb66c004bf0 con 0x7fb6800afa60 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:13.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb66c004f00 con 0x7fb6800afa60 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 monclient: dump: 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-29T11:32:13.121 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.122 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.122 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67effd640 4 set_mon_vals no callback set 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb66c0053f0 con 0x7fb6800afa60 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4280043783 0 0) 0x7fb66c006b40 con 0x7fb6800afa60 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 10 client.4150.objecter ms_dispatch 0x7fb680000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:32:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb67d7fa640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.067+0000 7fb687055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.068+0000 7fb687055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.068+0000 7fb687055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.068+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb660005080 con 0x7fb6800afa60 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.069+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb66c006590 con 0x7fb6800afa60 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.069+0000 7fb67d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.069+0000 7fb67d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:13.125 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.097+0000 7f2bbb020040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:13.125 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.081+0000 7fe430cc7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:32:13.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.142+0000 7fb687055640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b3d19353-b793-4b49-9361-aaf623b141ea", "id": 7}] 2024-03-29T11:32:13.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.142+0000 7fb687055640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b3d19353-b793-4b49-9361-aaf623b141ea", "id": 7}] 2024-03-29T11:32:13.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.142+0000 7fb687055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:13.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.142+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b3d19353-b793-4b49-9361-aaf623b141ea", "id": 7} v 0) v1 -- 0x7fb6600058b0 con 0x7fb6800afa60 2024-03-29T11:32:13.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.145+0000 7fb67d7fa640 1 -- 172.21.15.190:0/1802931244 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b3d19353-b793-4b49-9361-aaf623b141ea", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 478806 0 1371665406) 0x7fb66c006800 con 0x7fb6800afa60 2024-03-29T11:32:13.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.145+0000 7fb67d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b3d19353-b793-4b49-9361-aaf623b141ea", "id": 7}] 2024-03-29T11:32:13.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.145+0000 7fb67d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:13.141 INFO:teuthology.orchestra.run.smithi190.stdout:7 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 10 librados: watch_flush enter 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 10 librados: watch_flush exit 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 10 monclient: shutdown 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 >> v1:172.21.15.190:6789/0 conn(0x7fb6800afa60 legacy=0x7fb6800b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 shutdown_connections 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.146+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 >> 172.21.15.190:0/1802931244 conn(0x7fb680066b00 msgr2=0x7fb6800b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.147+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 shutdown_connections 2024-03-29T11:32:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.147+0000 7fb687055640 1 -- 172.21.15.190:0/1802931244 wait complete. 2024-03-29T11:32:13.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-29T11:32:13.147+0000 7fb687055640 1 librados: shutdown 2024-03-29T11:32:13.152 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:32:13.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:32:13.154 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:32:13.155 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:32:13.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:32:13.156 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:32:13.157 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:32:13.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:32:13.161 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:32:13.161 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:32:13.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:32:13.163 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:32:13.163 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:32:13.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:32:13.165 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:32:13.166 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:32:13.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:32:13.168 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:32:13.168 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:32:13.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:32:13.170 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:32:13.170 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:32:13.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:32:13.172 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:32:13.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:13.232 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.237+0000 7fe430cc7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:32:13.254 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.259+0000 7f2bbb020040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:32:13.287 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.292+0000 7fe430cc7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 10 monclient: get_monmap_and_config 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 10 monclient: build_initial_monmap 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 10 monclient: monmap: 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.297 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:32:13.303464+0000 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:32:13.303464+0000 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:13.298 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.304 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.305+0000 7f2bbb020040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:32:13.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 1 Processor -- start 2024-03-29T11:32:13.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 1 -- start start 2024-03-29T11:32:13.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.302+0000 7f18e52c5640 10 monclient: init 2024-03-29T11:32:13.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:13.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:13.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-b con 0x7f18e009ac40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:13.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-a con 0x7f18e013d800 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:13.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18de7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18e013d800 0x7f18e013fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46402/0 (socket says 172.21.15.45:46402) 2024-03-29T11:32:13.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18de7fc640 1 -- 172.21.15.45:0/2418091127 learned_addr learned my addr 172.21.15.45:0/2418091127 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:13.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-c con 0x7f18e009b020 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:13.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00aff40 con 0x7f18e013d800 2024-03-29T11:32:13.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b1740 con 0x7f18e009b020 2024-03-29T11:32:13.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b18d0 con 0x7f18e009ac40 2024-03-29T11:32:13.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:13.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18e52c5640 10 monclient(hunting): authenticate will time out at 789.542972s 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.303+0000 7f18deffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f18e009ac40 0x7f18e013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48048/0 (socket says 172.21.15.45:48048) 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018446436 0 0) 0x7f18e00aff40 con 0x7f18e013d800 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c8003520 con 0x7f18e013d800 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032009685 0 0) 0x7f18e00b18d0 con 0x7f18e009ac40 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e00aff40 con 0x7f18e009ac40 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698091850 0 0) 0x7f18e00b1740 con 0x7f18e009b020 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e00b18d0 con 0x7f18e009b020 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664970787 0 0) 0x7f18e00aff40 con 0x7f18e009ac40 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 >> v1:172.21.15.190:6789/0 conn(0x7f18e009b020 legacy=0x7f18e01405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 >> v1:172.21.15.45:6789/0 conn(0x7f18e013d800 legacy=0x7f18e013fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 monclient: found mon.noname-b 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00b1a60 con 0x7f18e009ac40 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:13.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d00015f0 con 0x7f18e009ac40 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: dump: 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.341 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18e52c5640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18e52c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d00017d0 con 0x7f18e009ac40 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 1 -- 172.21.15.45:0/2418091127 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d0001c20 con 0x7f18e009ac40 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18ddffb640 10 monclient: dump: 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.342 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18e52c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.304+0000 7f18e52c5640 4 set_mon_vals no callback set 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 10 monclient: shutdown 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 >> v1:172.21.15.45:6790/0 conn(0x7f18e009ac40 legacy=0x7f18e013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 shutdown_connections 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 >> 172.21.15.45:0/2418091127 conn(0x7f18e00b58f0 msgr2=0x7f18e00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 shutdown_connections 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.305+0000 7f18e52c5640 1 -- 172.21.15.45:0/2418091127 wait complete. 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.306+0000 7f18e52c5640 10 monclient: build_initial_monmap 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.306+0000 7f18e52c5640 10 monclient: monmap: 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.306+0000 7f18e52c5640 1 librados: starting msgr at 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.306+0000 7f18e52c5640 1 librados: starting objecter 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.306+0000 7f18e52c5640 1 Processor -- start 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 -- start start 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 librados: setting wanted keys 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 librados: calling monclient init 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient: init 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:13.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-b con 0x7f18e009ac40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-a con 0x7f18e009b020 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient(hunting): picked mon.noname-c con 0x7f18e013d800 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e0131980 con 0x7f18e009b020 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b37c0 con 0x7f18e009ac40 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00b3f30 con 0x7f18e013d800 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18e52c5640 10 monclient(hunting): authenticate will time out at 789.546917s 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18de7fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f18e009b020 0x7f18e01d2d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46404/0 (socket says 172.21.15.45:46404) 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.307+0000 7f18de7fc640 1 -- 172.21.15.45:0/720910234 learned_addr learned my addr 172.21.15.45:0/720910234 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e009b020 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e013d800 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter ms_handle_connect 0x7f18e009ac40 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206119261 0 0) 0x7f18e0131980 con 0x7f18e009b020 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc004eb0 con 0x7f18e009b020 2024-03-29T11:32:13.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123867553 0 0) 0x7f18e00b3f30 con 0x7f18e013d800 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0131980 con 0x7f18e013d800 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088136245 0 0) 0x7f18e00b37c0 con 0x7f18e009ac40 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:13.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e00b3f30 con 0x7f18e009ac40 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953162808 0 0) 0x7f18e0131980 con 0x7f18e013d800 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e00b37c0 con 0x7f18e013d800 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d8001bf0 con 0x7f18e013d800 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): dump: 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.349 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522943376 0 0) 0x7f18e00b3f30 con 0x7f18e009ac40 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e0131980 con 0x7f18e009ac40 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541313649 0 0) 0x7f18cc004eb0 con 0x7f18e009b020 2024-03-29T11:32:13.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18e00b3f30 con 0x7f18e009b020 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d0001ae0 con 0x7f18e009ac40 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): dump: 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.351 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.352 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.357 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d40017e0 con 0x7f18e009b020 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient(hunting): dump: 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.358 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 89250216 0 0) 0x7f18e00b37c0 con 0x7f18e013d800 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 >> v1:172.21.15.190:6789/0 conn(0x7f18e009ac40 legacy=0x7f18e013eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 >> v1:172.21.15.45:6789/0 conn(0x7f18e009b020 legacy=0x7f18e01d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 monclient: found mon.c 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e013f1f0 con 0x7f18e013d800 2024-03-29T11:32:13.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 10 monclient: _renew_subs 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e01316a0 con 0x7f18e013d800 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 10 monclient: _renew_subs 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e01d34a0 con 0x7f18e013d800 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d8002080 con 0x7f18e013d800 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:13.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f18d8003b30 con 0x7f18e013d800 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 10 monclient: dump: 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:13.361 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:13.362 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:13.362 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:13.362 INFO:teuthology.orchestra.run.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:13.378 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:32:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f18d8004060 con 0x7f18e013d800 2024-03-29T11:32:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18e52c5640 1 librados: init done 2024-03-29T11:32:13.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.309+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f18d80049e0 con 0x7f18e013d800 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.308+0000 7f18dd7fa640 4 set_mon_vals no callback set 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.309+0000 7f18c77fe640 10 client.4184.objecter ms_dispatch 0x7f18e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.309+0000 7f18c77fe640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.309+0000 7f18c77fe640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.309+0000 7f18c77fe640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.310+0000 7f18e52c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.310+0000 7f18e52c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.310+0000 7f18e52c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.310+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e004f730 con 0x7f18e013d800 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.312+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f18d8004c60 con 0x7f18e013d800 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.312+0000 7f18c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:13.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.312+0000 7f18c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:13.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.406+0000 7f18e52c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:13.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.406+0000 7f18e52c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.406+0000 7f18e52c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.406+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18e006cd40 con 0x7f18e013d800 2024-03-29T11:32:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.407+0000 7f18c77fe640 1 -- 172.21.15.45:0/720910234 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f18d8004310 con 0x7f18e013d800 2024-03-29T11:32:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.407+0000 7f18c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:13.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.407+0000 7f18c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:13.404 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-29T11:32:13.404 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.407+0000 7f18e52c5640 10 librados: watch_flush enter 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 10 librados: watch_flush exit 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 10 monclient: shutdown 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 >> v1:172.21.15.45:6790/0 conn(0x7f18e013d800 legacy=0x7f18e01dd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 shutdown_connections 2024-03-29T11:32:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 >> 172.21.15.45:0/720910234 conn(0x7f18e00b58f0 msgr2=0x7f18e00b5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 shutdown_connections 2024-03-29T11:32:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 -- 172.21.15.45:0/720910234 wait complete. 2024-03-29T11:32:13.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-29T11:32:13.408+0000 7f18e52c5640 1 librados: shutdown 2024-03-29T11:32:13.410 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.415+0000 7f2bbb020040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:13.413 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:32:13.414 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:32:13.434 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.439+0000 7fe430cc7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:13.506 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.509+0000 7f2bbb020040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:13.533 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.538+0000 7fe430cc7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:32:13.658 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.663+0000 7f2bbb020040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:13.693 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.698+0000 7fe430cc7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:32:13.699 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.704+0000 7f2bbb020040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:13.737 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.742+0000 7fe430cc7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:32:13.762 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-03-29T11:32:13.766+0000 7fd3db611640 -1 Falling back to public interface 2024-03-29T11:32:13.765 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-03-29T11:32:13.770+0000 7fd381e15640 -1 Falling back to public interface 2024-03-29T11:32:13.768 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-03-29T11:32:13.773+0000 7fe1a9e1f640 -1 Falling back to public interface 2024-03-29T11:32:13.768 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-03-29T11:32:13.773+0000 7fda2a610640 -1 Falling back to public interface 2024-03-29T11:32:13.774 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-03-29T11:32:13.779+0000 7f9184ad7640 -1 Falling back to public interface 2024-03-29T11:32:13.775 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-29T11:32:13.779+0000 7fa13580f640 -1 Falling back to public interface 2024-03-29T11:32:13.775 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-29T11:32:13.779+0000 7fa13421d640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:32:13.775 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-03-29T11:32:13.779+0000 7f9183a86640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T11:32:13.775 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-03-29T11:32:13.779+0000 7fd16bc0e640 -1 Falling back to public interface 2024-03-29T11:32:13.776 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-29T11:32:13.780+0000 7fa13421d640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6803/0: (98) Address already in use 2024-03-29T11:32:13.776 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-03-29T11:32:13.781+0000 7f75769f9640 -1 Falling back to public interface 2024-03-29T11:32:13.857 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:13.862+0000 7f2bbb020040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:32:13.904 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:13.906+0000 7fe430cc7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:32:14.219 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:14.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.314+0000 7fcf99702640 10 monclient: get_monmap_and_config 2024-03-29T11:32:14.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.314+0000 7fcf99702640 10 monclient: build_initial_monmap 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.314+0000 7fcf99702640 10 monclient: monmap: 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:14.315759+0000 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:14.315759+0000 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:14.310 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:14.311 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:14.311 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 1 Processor -- start 2024-03-29T11:32:14.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 1 -- start start 2024-03-29T11:32:14.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient: init 2024-03-29T11:32:14.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:14.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-b con 0x7fcf941457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-c con 0x7fcf9413f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-a con 0x7fcf940a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.315+0000 7fcf99702640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940aaef0 con 0x7fcf940a9d70 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf94145bd0 con 0x7fcf9413f650 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf94145eb0 con 0x7fcf941457f0 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 10 monclient(hunting): authenticate will time out at 790.555126s 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf92ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fcf941457f0 0x7fcf9413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48070/0 (socket says 172.21.15.45:48070) 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf92ffd640 1 -- 172.21.15.45:0/4220496903 learned_addr learned my addr 172.21.15.45:0/4220496903 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:14.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf937fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcf940a9d70 0x7fcf9414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:46406/0 (socket says 172.21.15.45:46406) 2024-03-29T11:32:14.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609244543 0 0) 0x7fcf94145eb0 con 0x7fcf941457f0 2024-03-29T11:32:14.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:32:14.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80003520 con 0x7fcf941457f0 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790132816 0 0) 0x7fcf940aaef0 con 0x7fcf940a9d70 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf94145eb0 con 0x7fcf940a9d70 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892499159 0 0) 0x7fcf94145bd0 con 0x7fcf9413f650 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf940aaef0 con 0x7fcf9413f650 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 26427750 0 0) 0x7fcf94145eb0 con 0x7fcf940a9d70 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 >> v1:172.21.15.45:6790/0 conn(0x7fcf941457f0 legacy=0x7fcf9413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 >> v1:172.21.15.190:6789/0 conn(0x7fcf9413f650 legacy=0x7fcf940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 monclient: found mon.noname-a 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94146340 con 0x7fcf940a9d70 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:14.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf8c0019f0 con 0x7fcf940a9d70 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: dump: 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.344 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c001b50 con 0x7fcf940a9d70 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 1 -- 172.21.15.45:0/4220496903 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf8c001fa0 con 0x7fcf940a9d70 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:14.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf927fc640 10 monclient: dump: 2024-03-29T11:32:14.346 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.346 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:14.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.316+0000 7fcf99702640 4 set_mon_vals no callback set 2024-03-29T11:32:14.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 10 monclient: shutdown 2024-03-29T11:32:14.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 1 -- 172.21.15.45:0/4220496903 >> v1:172.21.15.45:6789/0 conn(0x7fcf940a9d70 legacy=0x7fcf9414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 1 -- 172.21.15.45:0/4220496903 shutdown_connections 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 1 -- 172.21.15.45:0/4220496903 >> 172.21.15.45:0/4220496903 conn(0x7fcf94066a00 msgr2=0x7fcf94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 1 -- 172.21.15.45:0/4220496903 shutdown_connections 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.317+0000 7fcf99702640 1 -- 172.21.15.45:0/4220496903 wait complete. 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.318+0000 7fcf99702640 10 monclient: build_initial_monmap 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.318+0000 7fcf99702640 10 monclient: monmap: 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:14.350 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 librados: starting msgr at 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 librados: starting objecter 2024-03-29T11:32:14.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 Processor -- start 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 -- start start 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 librados: setting wanted keys 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 1 librados: calling monclient init 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.319+0000 7fcf99702640 10 monclient: init 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-c con 0x7fcf940a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:14.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9413f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient(hunting): picked mon.noname-b con 0x7fcf941457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940abf80 con 0x7fcf9413f650 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940ad8d0 con 0x7fcf941457f0 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940adbb0 con 0x7fcf940a9d70 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf99702640 10 monclient(hunting): authenticate will time out at 790.559337s 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf92ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fcf940a9d70 0x7fcf940ab870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48078/0 (socket says 172.21.15.45:48078) 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf92ffd640 1 -- 172.21.15.45:0/267832193 learned_addr learned my addr 172.21.15.45:0/267832193 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:14.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf940a9d70 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf9413f650 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf941457f0 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219752408 0 0) 0x7fcf940abf80 con 0x7fcf9413f650 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74004eb0 con 0x7fcf9413f650 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993811195 0 0) 0x7fcf940ad8d0 con 0x7fcf941457f0 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf940abf80 con 0x7fcf941457f0 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85955465 0 0) 0x7fcf940adbb0 con 0x7fcf940a9d70 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:14.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf940ad8d0 con 0x7fcf940a9d70 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 133838946 0 0) 0x7fcf940abf80 con 0x7fcf941457f0 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf940adbb0 con 0x7fcf941457f0 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf8c0019b0 con 0x7fcf941457f0 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.356 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:14.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651298972 0 0) 0x7fcf74004eb0 con 0x7fcf9413f650 2024-03-29T11:32:14.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf940abf80 con 0x7fcf9413f650 2024-03-29T11:32:14.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653840814 0 0) 0x7fcf940ad8d0 con 0x7fcf940a9d70 2024-03-29T11:32:14.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf74004eb0 con 0x7fcf940a9d70 2024-03-29T11:32:14.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf7c001870 con 0x7fcf9413f650 2024-03-29T11:32:14.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.320+0000 7fcf8b7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf84001790 con 0x7fcf940a9d70 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 123973074 0 0) 0x7fcf940adbb0 con 0x7fcf941457f0 2024-03-29T11:32:14.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 >> v1:172.21.15.45:6790/0 conn(0x7fcf940a9d70 legacy=0x7fcf940ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 >> v1:172.21.15.45:6789/0 conn(0x7fcf9413f650 legacy=0x7fcf94140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 monclient: found mon.b 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94138cb0 con 0x7fcf941457f0 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:14.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 5 monclient: authenticate success, global_id 4186 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 10 monclient: _renew_subs 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf941e2de0 con 0x7fcf941457f0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c004e10 con 0x7fcf941457f0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcf8c005120 con 0x7fcf941457f0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 10 monclient: _renew_subs 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf941e3150 con 0x7fcf941457f0 2024-03-29T11:32:14.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf99702640 1 librados: init done 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf917fa640 4 set_mon_vals no callback set 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 monclient: dump: 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:14.363 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcf8c005610 con 0x7fcf941457f0 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fcf8c005f90 con 0x7fcf941457f0 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 10 client.4186.objecter ms_dispatch 0x7fcf94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.321+0000 7fcf8b7fe640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.322+0000 7fcf99702640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.322+0000 7fcf99702640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.322+0000 7fcf99702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.322+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf94075f10 con 0x7fcf941457f0 2024-03-29T11:32:14.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.324+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcf8c0058c0 con 0x7fcf941457f0 2024-03-29T11:32:14.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.324+0000 7fcf8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:14.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.324+0000 7fcf8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.386+0000 7fcf99702640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.386+0000 7fcf99702640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.386+0000 7fcf99702640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.386+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf940b3680 con 0x7fcf941457f0 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.387+0000 7fcf8b7fe640 1 -- 172.21.15.45:0/267832193 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fcf8c005ae0 con 0x7fcf941457f0 2024-03-29T11:32:14.383 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.387+0000 7fcf8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:14.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.387+0000 7fcf8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:14.384 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:14.384 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:14.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 10 librados: watch_flush enter 2024-03-29T11:32:14.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 10 librados: watch_flush exit 2024-03-29T11:32:14.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 10 client.4186.objecter successfully canceled tick 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 10 monclient: shutdown 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 >> v1:172.21.15.190:6789/0 conn(0x7fcf941457f0 legacy=0x7fcf94141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 shutdown_connections 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 >> 172.21.15.45:0/267832193 conn(0x7fcf94066a00 msgr2=0x7fcf940b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 shutdown_connections 2024-03-29T11:32:14.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 -- 172.21.15.45:0/267832193 wait complete. 2024-03-29T11:32:14.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:14.388+0000 7fcf99702640 1 librados: shutdown 2024-03-29T11:32:14.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:14.674 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:14.676+0000 7f2bbb020040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:14.747 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:14.752+0000 7fe430cc7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:32:14.943 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:14.948+0000 7f2bbb020040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:32:14.993 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:14.999+0000 7f2bbb020040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:32:15.034 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:15.039+0000 7fe430cc7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:32:15.087 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:15.092+0000 7fe430cc7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:32:15.107 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:15.112+0000 7f2bbb020040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:32:15.210 INFO:tasks.ceph.mgr.y.smithi045.stderr:2024-03-29T11:32:15.215+0000 7fe430cc7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:32:16.122 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:16.126+0000 7f2b5b9fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:32:16.227 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-29T11:32:16.126+0000 7f2b5b9fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:32:17.227 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-03-29T11:32:17.232+0000 7fe1a9e1f640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:32:17.271 INFO:tasks.ceph.osd.0.smithi045.stderr:2024-03-29T11:32:17.275+0000 7fd3db611640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:32:17.292 INFO:tasks.ceph.osd.2.smithi045.stderr:2024-03-29T11:32:17.297+0000 7fd381e15640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:32:17.293 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-03-29T11:32:17.297+0000 7fda2a610640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:32:17.303 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-03-29T11:32:17.308+0000 7fd16bc0e640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:32:17.306 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-03-29T11:32:17.311+0000 7f9184ad7640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:32:17.307 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-29T11:32:17.312+0000 7fa13580f640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:32:17.310 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-03-29T11:32:17.315+0000 7f75769f9640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:32:21.201 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.353+0000 7fdde450e640 10 monclient: get_monmap_and_config 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.353+0000 7fdde450e640 10 monclient: build_initial_monmap 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.353+0000 7fdde450e640 10 monclient: monmap: 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:21.354369+0000 2024-03-29T11:32:21.349 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:21.354369+0000 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.354+0000 7fdde450e640 1 Processor -- start 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.354+0000 7fdde450e640 1 -- start start 2024-03-29T11:32:21.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.354+0000 7fdde450e640 10 monclient: init 2024-03-29T11:32:21.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-b con 0x7fdddc0b4100 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-a con 0x7fdddc13ee90 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-c con 0x7fdddc0a9ac0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0af6f0 con 0x7fdddc13ee90 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0aac80 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0aaf60 con 0x7fdddc0b4100 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:21.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.356+0000 7fdde450e640 10 monclient(hunting): authenticate will time out at 797.595704s 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde2aab640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdddc0a9ac0 0x7fdddc142ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:52222/0 (socket says 172.21.15.45:52222) 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde2aab640 1 -- 172.21.15.45:0/82486221 learned_addr learned my addr 172.21.15.45:0/82486221 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193970174 0 0) 0x7fdddc0aac80 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 10 monclient(hunting): _negotiate 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:32:21.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.357+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8003520 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 990910010 0 0) 0x7fddc8003520 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 >> v1:172.21.15.45:6790/0 conn(0x7fdddc0b4100 legacy=0x7fdddc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 >> v1:172.21.15.45:6789/0 conn(0x7fdddc13ee90 legacy=0x7fdddc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: _finish_hunting 0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 monclient: found mon.noname-c 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc13e970 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: _finish_auth 0 2024-03-29T11:32:21.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: _check_auth_tickets 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddd40019f0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 10 monclient: dump: 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.355 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.356 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde450e640 5 monclient: authenticate success, global_id 4225 2024-03-29T11:32:21.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde450e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.358+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd4001b50 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde12a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde12a8640 1 -- 172.21.15.45:0/82486221 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddd4001fa0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde12a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde12a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde12a8640 10 monclient: dump: 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.357 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde450e640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.359+0000 7fdde450e640 4 set_mon_vals no callback set 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 10 monclient: shutdown 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:21.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 1 -- 172.21.15.45:0/82486221 >> v1:172.21.15.190:6789/0 conn(0x7fdddc0a9ac0 legacy=0x7fdddc142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 1 -- 172.21.15.45:0/82486221 shutdown_connections 2024-03-29T11:32:21.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 1 -- 172.21.15.45:0/82486221 >> 172.21.15.45:0/82486221 conn(0x7fdddc0b35e0 msgr2=0x7fdddc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:21.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 1 -- 172.21.15.45:0/82486221 shutdown_connections 2024-03-29T11:32:21.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.360+0000 7fdde450e640 1 -- 172.21.15.45:0/82486221 wait complete. 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.361+0000 7fdde450e640 10 monclient: build_initial_monmap 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.361+0000 7fdde450e640 10 monclient: monmap: 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:21.361 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 librados: starting msgr at 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 librados: starting objecter 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 Processor -- start 2024-03-29T11:32:21.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 -- start start 2024-03-29T11:32:21.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 librados: setting wanted keys 2024-03-29T11:32:21.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 1 librados: calling monclient init 2024-03-29T11:32:21.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.362+0000 7fdde450e640 10 monclient: init 2024-03-29T11:32:21.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:21.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-b con 0x7fdddc0a9ac0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-c con 0x7fdddc0b4100 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient(hunting): picked mon.noname-a con 0x7fdddc13ee90 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc13fcd0 con 0x7fdddc13ee90 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc140e20 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc13d8d0 con 0x7fdddc0b4100 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.363+0000 7fdde450e640 10 monclient(hunting): authenticate will time out at 797.602716s 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fdde22aa640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdddc0a9ac0 0x7fdddc14e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:52346/0 (socket says 172.21.15.45:52346) 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fdde22aa640 1 -- 172.21.15.45:0/345241446 learned_addr learned my addr 172.21.15.45:0/345241446 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter ms_handle_connect 0x7fdddc0a9ac0 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter ms_handle_connect 0x7fdddc13ee90 2024-03-29T11:32:21.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter ms_handle_connect 0x7fdddc0b4100 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551193978 0 0) 0x7fdddc13fcd0 con 0x7fdddc13ee90 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.364+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0004eb0 con 0x7fdddc13ee90 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386732618 0 0) 0x7fdddc140e20 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:32:21.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdddc13fcd0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785466026 0 0) 0x7fdddc13d8d0 con 0x7fdddc0b4100 2024-03-29T11:32:21.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:21.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:21.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdddc140e20 con 0x7fdddc0b4100 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3291913776 0 0) 0x7fddc0004eb0 con 0x7fdddc13ee90 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.365+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdddc13d8d0 con 0x7fdddc13ee90 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddd4001e80 con 0x7fdddc13ee90 2024-03-29T11:32:21.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:21.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2109860844 0 0) 0x7fdddc13fcd0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc0004eb0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3836681464 0 0) 0x7fdddc140e20 con 0x7fdddc0b4100 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdddc13fcd0 con 0x7fdddc0b4100 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddd80019b0 con 0x7fdddc0a9ac0 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.374 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddcc0016d0 con 0x7fdddc0b4100 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.366+0000 7fddd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 10 monclient(hunting): dump: 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.375 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2237213025 0 0) 0x7fdddc13d8d0 con 0x7fdddc13ee90 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 >> v1:172.21.15.45:6790/0 conn(0x7fdddc0b4100 legacy=0x7fdddc13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 >> v1:172.21.15.190:6789/0 conn(0x7fdddc0a9ac0 legacy=0x7fdddc14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 1 monclient: found mon.a 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.367+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc11f3a0 con 0x7fdddc13ee90 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fddd2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:21.376 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fddd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fddd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 10 monclient: _renew_subs 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc12c140 con 0x7fdddc13ee90 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 10 monclient: _renew_subs 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc1508e0 con 0x7fdddc13ee90 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.368+0000 7fdde450e640 1 librados: init done 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd4001fe0 con 0x7fdddc13ee90 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fddd4004fb0 con 0x7fdddc13ee90 2024-03-29T11:32:21.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fddd2ffd640 10 monclient: dump: 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:21.378 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.369+0000 7fdde0aa7640 4 set_mon_vals no callback set 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.370+0000 7fdde450e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.370+0000 7fdde450e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.370+0000 7fdde450e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.371+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc4005080 con 0x7fdddc13ee90 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.372+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fddd401af80 con 0x7fdddc13ee90 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.372+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fddd4084e40 con 0x7fdddc13ee90 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fddd2ffd640 10 client.4245.objecter ms_dispatch 0x7fdddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fddd2ffd640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fdde22aa640 10 monclient: get_auth_request con 0x7fddc006e070 auth_method 0 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fddd2ffd640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fddd2ffd640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:32:21.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.373+0000 7fddd2ffd640 10 client.4245.objecter ms_handle_connect 0x7fddc006e070 2024-03-29T11:32:21.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.384+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddd4050720 con 0x7fdddc13ee90 2024-03-29T11:32:21.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.384+0000 7fddd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:21.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.384+0000 7fddd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.756+0000 7fdde450e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.756+0000 7fdde450e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.756+0000 7fdde450e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:21.752 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.756+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddc4005330 con 0x7fdddc13ee90 2024-03-29T11:32:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.758+0000 7fddd2ffd640 1 -- 172.21.15.45:0/345241446 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fddd4052650 con 0x7fdddc13ee90 2024-03-29T11:32:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.758+0000 7fddd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:21.753 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.758+0000 7fddd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:21.754 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:21.754 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:21.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.762+0000 7fdde450e640 10 librados: watch_flush enter 2024-03-29T11:32:21.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.762+0000 7fdde450e640 10 librados: watch_flush exit 2024-03-29T11:32:21.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:32:21.758 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:32:21.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 >> v1:172.21.15.190:6816/1411598462 conn(0x7fddc006e070 legacy=0x7fddc0070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 10 monclient: shutdown 2024-03-29T11:32:21.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:21.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 >> v1:172.21.15.45:6789/0 conn(0x7fdddc13ee90 legacy=0x7fdddc1e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:21.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 shutdown_connections 2024-03-29T11:32:21.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 >> 172.21.15.45:0/345241446 conn(0x7fdddc0b35e0 msgr2=0x7fdddc0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:21.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.763+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 shutdown_connections 2024-03-29T11:32:21.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.764+0000 7fdde450e640 1 -- 172.21.15.45:0/345241446 wait complete. 2024-03-29T11:32:21.760 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:21.764+0000 7fdde450e640 1 librados: shutdown 2024-03-29T11:32:21.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:28.581 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.734+0000 7fa3fa966640 10 monclient: get_monmap_and_config 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.734+0000 7fa3fa966640 10 monclient: build_initial_monmap 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.734+0000 7fa3fa966640 10 monclient: monmap: 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:28.735641+0000 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:28.735641+0000 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.730 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:28.731 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:28.731 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:28.731 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.735+0000 7fa3fa966640 1 Processor -- start 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.735+0000 7fa3fa966640 1 -- start start 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.735+0000 7fa3fa966640 10 monclient: init 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ec0389e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec038dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ec030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec0c54c0 con 0x7fa3ec0389e0 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec036430 con 0x7fa3ec038dc0 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec039840 con 0x7fa3ec030660 2024-03-29T11:32:28.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:28.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.736+0000 7fa3fa966640 10 monclient(hunting): authenticate will time out at 804.975930s 2024-03-29T11:32:28.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f9964640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa3ec0389e0 0x7fa3ec033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52428/0 (socket says 172.21.15.45:52428) 2024-03-29T11:32:28.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f9964640 1 -- 172.21.15.45:0/2989791030 learned_addr learned my addr 172.21.15.45:0/2989791030 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:28.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906834316 0 0) 0x7fa3ec039840 con 0x7fa3ec030660 2024-03-29T11:32:28.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e0003520 con 0x7fa3ec030660 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7704655 0 0) 0x7fa3ec036430 con 0x7fa3ec038dc0 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.737+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec039840 con 0x7fa3ec038dc0 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915671369 0 0) 0x7fa3ec0c54c0 con 0x7fa3ec0389e0 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:32:28.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec036430 con 0x7fa3ec0389e0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320431007 0 0) 0x7fa3ec039840 con 0x7fa3ec038dc0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 >> v1:172.21.15.45:6790/0 conn(0x7fa3ec030660 legacy=0x7fa3ec0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 >> v1:172.21.15.45:6789/0 conn(0x7fa3ec0389e0 legacy=0x7fa3ec033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: _finish_hunting 0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 monclient: found mon.noname-c 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec035690 con 0x7fa3ec038dc0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: _finish_auth 0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: _check_auth_tickets 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3fa966640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa3f0001690 con 0x7fa3ec038dc0 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3fa966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:28.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3f8962640 10 monclient: dump: 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.738+0000 7fa3fa966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:28.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3f00017f0 con 0x7fa3ec038dc0 2024-03-29T11:32:28.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:28.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3fa966640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:28.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3fa966640 4 set_mon_vals no callback set 2024-03-29T11:32:28.746 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 1 -- 172.21.15.45:0/2989791030 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa3f0001c00 con 0x7fa3ec038dc0 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.739+0000 7fa3f8962640 10 monclient: dump: 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:28.834 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.740+0000 7fa3fa966640 10 monclient: shutdown 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.740+0000 7fa3fa966640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.740+0000 7fa3fa966640 1 -- 172.21.15.45:0/2989791030 >> v1:172.21.15.190:6789/0 conn(0x7fa3ec038dc0 legacy=0x7fa3ec0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.740+0000 7fa3fa966640 1 -- 172.21.15.45:0/2989791030 shutdown_connections 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.740+0000 7fa3fa966640 1 -- 172.21.15.45:0/2989791030 >> 172.21.15.45:0/2989791030 conn(0x7fa3ec005c00 msgr2=0x7fa3ec006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.741+0000 7fa3fa966640 1 -- 172.21.15.45:0/2989791030 shutdown_connections 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.741+0000 7fa3fa966640 1 -- 172.21.15.45:0/2989791030 wait complete. 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.741+0000 7fa3fa966640 10 monclient: build_initial_monmap 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.741+0000 7fa3fa966640 10 monclient: monmap: 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:28.835 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 librados: starting msgr at 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 librados: starting objecter 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 Processor -- start 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 -- start start 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 librados: setting wanted keys 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 1 librados: calling monclient init 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.742+0000 7fa3fa966640 10 monclient: init 2024-03-29T11:32:28.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ec030660 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec0389e0 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ec0c84d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec0c0cb0 con 0x7fa3ec030660 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec0301b0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec02f030 con 0x7fa3ec0389e0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa966640 10 monclient(hunting): authenticate will time out at 804.982555s 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa165640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa3ec0c84d0 0x7fa3ec164990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:35402/0 (socket says 172.21.15.45:35402) 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3fa165640 1 -- 172.21.15.45:0/3069366382 learned_addr learned my addr 172.21.15.45:0/3069366382 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.743+0000 7fa3e67fc640 10 client.?.objecter ms_handle_connect 0x7fa3ec0c84d0 2024-03-29T11:32:28.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 client.?.objecter ms_handle_connect 0x7fa3ec030660 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 client.?.objecter ms_handle_connect 0x7fa3ec0389e0 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823770867 0 0) 0x7fa3ec0301b0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d8004eb0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800553026 0 0) 0x7fa3ec0c0cb0 con 0x7fa3ec030660 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.838 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec0301b0 con 0x7fa3ec030660 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452176669 0 0) 0x7fa3ec02f030 con 0x7fa3ec0389e0 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec0c0cb0 con 0x7fa3ec0389e0 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3883385470 0 0) 0x7fa3ec0301b0 con 0x7fa3ec030660 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ec02f030 con 0x7fa3ec030660 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa3e80017e0 con 0x7fa3ec030660 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): dump: 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:28.839 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4137495892 0 0) 0x7fa3d8004eb0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ec0301b0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa3f404fdd0 con 0x7fa3ec0c84d0 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:28.840 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.744+0000 7fa3e67fc640 10 monclient(hunting): dump: 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082195070 0 0) 0x7fa3ec02f030 con 0x7fa3ec030660 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 >> v1:172.21.15.45:6790/0 conn(0x7fa3ec0389e0 legacy=0x7fa3ec0bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 >> v1:172.21.15.190:6789/0 conn(0x7fa3ec0c84d0 legacy=0x7fa3ec164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:28.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 1 monclient: found mon.a 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec0c0f00 con 0x7fa3ec030660 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 10 monclient: _renew_subs 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ec0c0190 con 0x7fa3ec030660 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 10 monclient: _renew_subs 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:28.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ec1650a0 con 0x7fa3ec030660 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.745+0000 7fa3fa966640 1 librados: init done 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.746+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e8001cd0 con 0x7fa3ec030660 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.746+0000 7fa3e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.746+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa3e8004980 con 0x7fa3ec030660 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.746+0000 7fa3e7fff640 4 set_mon_vals no callback set 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.747+0000 7fa3e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.747+0000 7fa3e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.747+0000 7fa3e67fc640 10 monclient: dump: 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:28.843 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.747+0000 7fa3fa966640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.748+0000 7fa3fa966640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.748+0000 7fa3fa966640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.748+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc005080 con 0x7fa3ec030660 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.749+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fa3e8004ae0 con 0x7fa3ec030660 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fa3e80849d0 con 0x7fa3ec030660 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 10 client.4251.objecter ms_dispatch 0x7fa3f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3f9163640 10 monclient: get_auth_request con 0x7fa3d806ded0 auth_method 0 2024-03-29T11:32:28.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:32:28.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.750+0000 7fa3e67fc640 10 client.4251.objecter ms_handle_connect 0x7fa3d806ded0 2024-03-29T11:32:28.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.758+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3e80502b0 con 0x7fa3ec030660 2024-03-29T11:32:28.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.758+0000 7fa3e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:28.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:28.758+0000 7fa3e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:29.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.131+0000 7fa3fa966640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:29.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.131+0000 7fa3fa966640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:29.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.131+0000 7fa3fa966640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:29.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.131+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3dc005330 con 0x7fa3ec030660 2024-03-29T11:32:29.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.133+0000 7fa3e67fc640 1 -- 172.21.15.45:0/3069366382 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fa3e80521e0 con 0x7fa3ec030660 2024-03-29T11:32:29.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.133+0000 7fa3e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:29.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.133+0000 7fa3e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:29.129 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:29.129 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.136+0000 7fa3fa966640 10 librados: watch_flush enter 2024-03-29T11:32:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.136+0000 7fa3fa966640 10 librados: watch_flush exit 2024-03-29T11:32:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.136+0000 7fa3fa966640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:32:29.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.136+0000 7fa3fa966640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 >> v1:172.21.15.190:6816/1411598462 conn(0x7fa3d806ded0 legacy=0x7fa3d8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 10 monclient: shutdown 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 >> v1:172.21.15.45:6789/0 conn(0x7fa3ec030660 legacy=0x7fa3ec02faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 shutdown_connections 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 >> 172.21.15.45:0/3069366382 conn(0x7fa3ec005c00 msgr2=0x7fa3ec034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 shutdown_connections 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 -- 172.21.15.45:0/3069366382 wait complete. 2024-03-29T11:32:29.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:29.137+0000 7fa3fa966640 1 librados: shutdown 2024-03-29T11:32:29.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:35.955 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:36.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.105+0000 7fba79263640 10 monclient: get_monmap_and_config 2024-03-29T11:32:36.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.105+0000 7fba79263640 10 monclient: build_initial_monmap 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.105+0000 7fba79263640 10 monclient: monmap: 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:36.106369+0000 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:36.106369+0000 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:36.101 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.107+0000 7fba79263640 1 Processor -- start 2024-03-29T11:32:36.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.107+0000 7fba79263640 1 -- start start 2024-03-29T11:32:36.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.107+0000 7fba79263640 10 monclient: init 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient(hunting): picked mon.noname-c con 0x7fba741457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient(hunting): picked mon.noname-a con 0x7fba7413f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient(hunting): picked mon.noname-b con 0x7fba740a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:36.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba740aaef0 con 0x7fba7413f650 2024-03-29T11:32:36.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74145bd0 con 0x7fba741457f0 2024-03-29T11:32:36.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74145eb0 con 0x7fba740a9d70 2024-03-29T11:32:36.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:36.104 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.108+0000 7fba79263640 10 monclient(hunting): authenticate will time out at 812.347476s 2024-03-29T11:32:36.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.112+0000 7fba72ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fba741457f0 0x7fba7413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:49450/0 (socket says 172.21.15.45:49450) 2024-03-29T11:32:36.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.112+0000 7fba72ffd640 1 -- 172.21.15.45:0/2683088106 learned_addr learned my addr 172.21.15.45:0/2683088106 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.112+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998224452 0 0) 0x7fba74145eb0 con 0x7fba740a9d70 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5c003520 con 0x7fba740a9d70 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567960488 0 0) 0x7fba5c003520 con 0x7fba740a9d70 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 >> v1:172.21.15.190:6789/0 conn(0x7fba741457f0 legacy=0x7fba7413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 >> v1:172.21.15.45:6789/0 conn(0x7fba7413f650 legacy=0x7fba740b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 monclient: found mon.noname-b 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:32:36.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74146340 con 0x7fba740a9d70 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fba68001690 con 0x7fba740a9d70 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: dump: 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.110 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba680018f0 con 0x7fba740a9d70 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 1 -- 172.21.15.45:0/2683088106 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fba68001690 con 0x7fba740a9d70 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.113+0000 7fba71ffb640 10 monclient: dump: 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.111 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:36.112 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.112 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:36.112 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:36.112 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:36.112 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.114+0000 7fba79263640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.114+0000 7fba79263640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.114+0000 7fba79263640 4 set_mon_vals no callback set 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 10 monclient: shutdown 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 1 -- 172.21.15.45:0/2683088106 >> v1:172.21.15.45:6790/0 conn(0x7fba740a9d70 legacy=0x7fba7414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 1 -- 172.21.15.45:0/2683088106 shutdown_connections 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 1 -- 172.21.15.45:0/2683088106 >> 172.21.15.45:0/2683088106 conn(0x7fba74066a00 msgr2=0x7fba74066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 1 -- 172.21.15.45:0/2683088106 shutdown_connections 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.115+0000 7fba79263640 1 -- 172.21.15.45:0/2683088106 wait complete. 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.116+0000 7fba79263640 10 monclient: build_initial_monmap 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.116+0000 7fba79263640 10 monclient: monmap: 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:36.115 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 librados: starting msgr at 2024-03-29T11:32:36.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 librados: starting objecter 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 Processor -- start 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 -- start start 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 librados: setting wanted keys 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 1 librados: calling monclient init 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.117+0000 7fba79263640 10 monclient: init 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient(hunting): picked mon.noname-a con 0x7fba740a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient(hunting): picked mon.noname-c con 0x7fba7413f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient(hunting): picked mon.noname-b con 0x7fba741457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74141420 con 0x7fba740a9d70 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74141700 con 0x7fba741457f0 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba74142c20 con 0x7fba7413f650 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:36.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.118+0000 7fba79263640 10 monclient(hunting): authenticate will time out at 812.358024s 2024-03-29T11:32:36.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.119+0000 7fba737fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fba741457f0 0x7fba74142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:49588/0 (socket says 172.21.15.45:49588) 2024-03-29T11:32:36.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.119+0000 7fba737fe640 1 -- 172.21.15.45:0/3317809619 learned_addr learned my addr 172.21.15.45:0/3317809619 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:36.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.119+0000 7fba5b7fe640 10 client.?.objecter ms_handle_connect 0x7fba741457f0 2024-03-29T11:32:36.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.119+0000 7fba5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:36.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 client.?.objecter ms_handle_connect 0x7fba7413f650 2024-03-29T11:32:36.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:36.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021995965 0 0) 0x7fba74141700 con 0x7fba741457f0 2024-03-29T11:32:36.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba60004eb0 con 0x7fba741457f0 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733788156 0 0) 0x7fba74142c20 con 0x7fba7413f650 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba60006230 con 0x7fba7413f650 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 client.?.objecter ms_handle_connect 0x7fba740a9d70 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.122+0000 7fba5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010938450 0 0) 0x7fba74141420 con 0x7fba740a9d70 2024-03-29T11:32:36.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba74142c20 con 0x7fba740a9d70 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081014322 0 0) 0x7fba74142c20 con 0x7fba740a9d70 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba74141420 con 0x7fba740a9d70 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fba6c001a70 con 0x7fba740a9d70 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.122 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 838063048 0 0) 0x7fba60006230 con 0x7fba7413f650 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba74142c20 con 0x7fba7413f650 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fba64001830 con 0x7fba7413f650 2024-03-29T11:32:36.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.123+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1437783690 0 0) 0x7fba74141420 con 0x7fba740a9d70 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 >> v1:172.21.15.45:6790/0 conn(0x7fba7413f650 legacy=0x7fba74141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 >> v1:172.21.15.190:6789/0 conn(0x7fba741457f0 legacy=0x7fba74142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 1 monclient: found mon.a 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7413bc40 con 0x7fba740a9d70 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 10 monclient: _renew_subs 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba74140850 con 0x7fba740a9d70 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:36.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 10 monclient: _renew_subs 2024-03-29T11:32:36.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:36.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7413c6d0 con 0x7fba740a9d70 2024-03-29T11:32:36.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.124+0000 7fba79263640 1 librados: init done 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.127+0000 7fba79263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba6c001f60 con 0x7fba740a9d70 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fba6c004d00 con 0x7fba740a9d70 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba5b7fe640 10 monclient: dump: 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:36.127 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:36.128 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:36.129 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:36.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.128+0000 7fba717fa640 4 set_mon_vals no callback set 2024-03-29T11:32:36.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.129+0000 7fba79263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:36.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.129+0000 7fba79263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:36.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.129+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7413c9d0 con 0x7fba740a9d70 2024-03-29T11:32:36.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fba6c01acd0 con 0x7fba740a9d70 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fba6c084930 con 0x7fba740a9d70 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 10 client.4254.objecter ms_dispatch 0x7fba74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.133+0000 7fba5b7fe640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.134+0000 7fba727fc640 10 monclient: get_auth_request con 0x7fba6006e110 auth_method 0 2024-03-29T11:32:36.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.134+0000 7fba5b7fe640 10 client.4254.objecter ms_handle_connect 0x7fba6006e110 2024-03-29T11:32:36.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.140+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba6c050350 con 0x7fba740a9d70 2024-03-29T11:32:36.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.140+0000 7fba5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:36.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.140+0000 7fba5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:36.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba79263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:36.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba79263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:36.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba79263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:36.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba74075f10 con 0x7fba740a9d70 2024-03-29T11:32:36.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba5b7fe640 1 -- 172.21.15.45:0/3317809619 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fba6c052280 con 0x7fba740a9d70 2024-03-29T11:32:36.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:36.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.468+0000 7fba5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:36.466 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:36.466 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:36.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 10 librados: watch_flush enter 2024-03-29T11:32:36.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 10 librados: watch_flush exit 2024-03-29T11:32:36.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T11:32:36.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 10 client.4254.objecter successfully canceled tick 2024-03-29T11:32:36.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 >> v1:172.21.15.190:6816/1411598462 conn(0x7fba6006e110 legacy=0x7fba600705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 10 monclient: shutdown 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 >> v1:172.21.15.45:6789/0 conn(0x7fba740a9d70 legacy=0x7fba74140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 shutdown_connections 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 >> 172.21.15.45:0/3317809619 conn(0x7fba74066a00 msgr2=0x7fba740b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 shutdown_connections 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.472+0000 7fba79263640 1 -- 172.21.15.45:0/3317809619 wait complete. 2024-03-29T11:32:36.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:36.473+0000 7fba79263640 1 librados: shutdown 2024-03-29T11:32:36.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:43.294 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:43.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.446+0000 7f90c52ae640 10 monclient: get_monmap_and_config 2024-03-29T11:32:43.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.446+0000 7f90c52ae640 10 monclient: build_initial_monmap 2024-03-29T11:32:43.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.446+0000 7f90c52ae640 10 monclient: monmap: 2024-03-29T11:32:43.442 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:43.442 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:43.447507+0000 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:43.447507+0000 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:43.443 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:43.444 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.447+0000 7f90c52ae640 1 Processor -- start 2024-03-29T11:32:43.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.447+0000 7f90c52ae640 1 -- start start 2024-03-29T11:32:43.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.447+0000 7f90c52ae640 10 monclient: init 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.448+0000 7f90c52ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.448+0000 7f90c52ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.448+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-a con 0x7f90c01457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.449+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-c con 0x7f90c013f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.449+0000 7f90b7fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f90c013f650 0x7f90c00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:47816/0 (socket says 172.21.15.45:47816) 2024-03-29T11:32:43.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.449+0000 7f90b7fff640 1 -- 172.21.15.45:0/526710278 learned_addr learned my addr 172.21.15.45:0/526710278 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.449+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-b con 0x7f90c00a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.449+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c00aaef0 con 0x7f90c01457f0 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c0145bd0 con 0x7f90c013f650 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c0145eb0 con 0x7f90c00a9d70 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90c52ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:43.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90c52ae640 10 monclient(hunting): authenticate will time out at 819.689216s 2024-03-29T11:32:43.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663013925 0 0) 0x7f90c0145bd0 con 0x7f90c013f650 2024-03-29T11:32:43.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:43.454 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ac003520 con 0x7f90c013f650 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635277855 0 0) 0x7f90ac003520 con 0x7f90c013f650 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.450+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 >> v1:172.21.15.45:6790/0 conn(0x7f90c00a9d70 legacy=0x7f90c014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 >> v1:172.21.15.45:6789/0 conn(0x7f90c01457f0 legacy=0x7f90c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.455 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 1 monclient: found mon.noname-c 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c0146340 con 0x7f90c013f650 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:43.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:43.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:43.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90a8001690 con 0x7f90c013f650 2024-03-29T11:32:43.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:43.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:43.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90be7fc640 10 monclient: dump: 2024-03-29T11:32:43.458 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:43.458 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:43.458 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.458 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.459 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:43.459 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.459 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:43.459 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:43.459 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:43.460 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90c52ae640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:32:43.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.452+0000 7f90c52ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:43.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a8001860 con 0x7f90c013f650 2024-03-29T11:32:43.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:43.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 1 -- 172.21.15.45:0/526710278 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90a8001c70 con 0x7f90c013f650 2024-03-29T11:32:43.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:43.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:43.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90be7fc640 10 monclient: dump: 2024-03-29T11:32:43.461 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:43.462 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:43.462 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.462 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.462 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:43.462 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90c52ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.453+0000 7f90c52ae640 4 set_mon_vals no callback set 2024-03-29T11:32:43.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.454+0000 7f90c52ae640 10 monclient: shutdown 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.454+0000 7f90c52ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.454+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 >> v1:172.21.15.190:6789/0 conn(0x7f90c013f650 legacy=0x7f90c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.454+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 shutdown_connections 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.454+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 >> 172.21.15.45:0/526710278 conn(0x7f90c0066a00 msgr2=0x7f90c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.455+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 shutdown_connections 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.455+0000 7f90c52ae640 1 -- 172.21.15.45:0/526710278 wait complete. 2024-03-29T11:32:43.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.456+0000 7f90c52ae640 10 monclient: build_initial_monmap 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.456+0000 7f90c52ae640 10 monclient: monmap: 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:43.465 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.456+0000 7f90c52ae640 1 librados: starting msgr at 2024-03-29T11:32:43.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.456+0000 7f90c52ae640 1 librados: starting objecter 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.457+0000 7f90c52ae640 1 Processor -- start 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.457+0000 7f90c52ae640 1 -- start start 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.457+0000 7f90c52ae640 1 librados: setting wanted keys 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.457+0000 7f90c52ae640 1 librados: calling monclient init 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.457+0000 7f90c52ae640 10 monclient: init 2024-03-29T11:32:43.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.458+0000 7f90c52ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:43.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.458+0000 7f90c52ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:43.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.458+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-b con 0x7f90c00a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:43.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.458+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-a con 0x7f90c013f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:43.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 10 monclient(hunting): picked mon.noname-c con 0x7f90c01457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:43.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c013ae00 con 0x7f90c013f650 2024-03-29T11:32:43.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c01d79c0 con 0x7f90c00a9d70 2024-03-29T11:32:43.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c00b1700 con 0x7f90c01457f0 2024-03-29T11:32:43.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90c52ae640 10 monclient(hunting): authenticate will time out at 819.699196s 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90beffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f90c00a9d70 0x7f90c01d7ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:48186/0 (socket says 172.21.15.45:48186) 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90beffd640 1 -- 172.21.15.45:0/3866361513 learned_addr learned my addr 172.21.15.45:0/3866361513 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90b77fe640 10 client.?.objecter ms_handle_connect 0x7f90c00a9d70 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:43.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.460+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030772503 0 0) 0x7f90c01d79c0 con 0x7f90c00a9d70 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0004eb0 con 0x7f90c00a9d70 2024-03-29T11:32:43.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.461+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884129204 0 0) 0x7f90a0004eb0 con 0x7f90c00a9d70 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90c01d79c0 con 0x7f90c00a9d70 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 client.?.objecter ms_handle_connect 0x7f90c013f650 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90b0001890 con 0x7f90c00a9d70 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:43.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): dump: 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.488 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060907668 0 0) 0x7f90c013ae00 con 0x7f90c013f650 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:32:43.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0004eb0 con 0x7f90c013f650 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.462+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125717067 0 0) 0x7f90c01d79c0 con 0x7f90c00a9d70 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 >> v1:172.21.15.45:6790/0 conn(0x7f90c01457f0 legacy=0x7f90c01e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 >> v1:172.21.15.45:6789/0 conn(0x7f90c013f650 legacy=0x7f90c013b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 1 monclient: found mon.b 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:43.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c0139aa0 con 0x7f90c00a9d70 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 10 monclient: _renew_subs 2024-03-29T11:32:43.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c01d75e0 con 0x7f90c00a9d70 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 10 monclient: _renew_subs 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.463+0000 7f90c52ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.464+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c0142d00 con 0x7f90c00a9d70 2024-03-29T11:32:43.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.464+0000 7f90c52ae640 1 librados: init done 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90b0001a60 con 0x7f90c00a9d70 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90b00038c0 con 0x7f90c00a9d70 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:43.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90bdffb640 4 set_mon_vals no callback set 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90b77fe640 10 monclient: dump: 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:43.495 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.465+0000 7f90c52ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:43.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.466+0000 7f90c52ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:43.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.466+0000 7f90c52ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.466+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a4005080 con 0x7f90c00a9d70 2024-03-29T11:32:43.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.470+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f90b0003a20 con 0x7f90c00a9d70 2024-03-29T11:32:43.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.471+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f90b00848f0 con 0x7f90c00a9d70 2024-03-29T11:32:43.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.471+0000 7f90b77fe640 10 client.4243.objecter ms_dispatch 0x7f90c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.471+0000 7f90b77fe640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.471+0000 7f90b77fe640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.471+0000 7f90b77fe640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.472+0000 7f90b7fff640 10 monclient: get_auth_request con 0x7f90a006dd80 auth_method 0 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.472+0000 7f90b77fe640 10 client.4243.objecter ms_handle_connect 0x7f90a006dd80 2024-03-29T11:32:43.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.475+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90b00501d0 con 0x7f90c00a9d70 2024-03-29T11:32:43.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.475+0000 7f90b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:43.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.475+0000 7f90b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:43.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.802+0000 7f90c52ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:43.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.802+0000 7f90c52ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:43.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.802+0000 7f90c52ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:32:43.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.802+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90a4005330 con 0x7f90c00a9d70 2024-03-29T11:32:43.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.803+0000 7f90b77fe640 1 -- 172.21.15.45:0/3866361513 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f90b0052100 con 0x7f90c00a9d70 2024-03-29T11:32:43.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.803+0000 7f90b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:43.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.803+0000 7f90b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:43.800 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:43.800 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:43.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 10 librados: watch_flush enter 2024-03-29T11:32:43.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 10 librados: watch_flush exit 2024-03-29T11:32:43.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:32:43.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:32:43.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 >> v1:172.21.15.190:6816/1411598462 conn(0x7f90a006dd80 legacy=0x7f90a0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 10 monclient: shutdown 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 >> v1:172.21.15.190:6789/0 conn(0x7f90c00a9d70 legacy=0x7f90c01d7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 shutdown_connections 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 >> 172.21.15.45:0/3866361513 conn(0x7f90c0066a00 msgr2=0x7f90c00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 shutdown_connections 2024-03-29T11:32:43.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 -- 172.21.15.45:0/3866361513 wait complete. 2024-03-29T11:32:43.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:43.807+0000 7f90c52ae640 1 librados: shutdown 2024-03-29T11:32:43.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:50.627 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:50.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.786+0000 7fbb66634640 10 monclient: get_monmap_and_config 2024-03-29T11:32:50.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.786+0000 7fbb66634640 10 monclient: build_initial_monmap 2024-03-29T11:32:50.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.786+0000 7fbb66634640 10 monclient: monmap: 2024-03-29T11:32:50.781 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:50.786308+0000 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:50.786308+0000 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.787+0000 7fbb66634640 1 Processor -- start 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.787+0000 7fbb66634640 1 -- start start 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.787+0000 7fbb66634640 10 monclient: init 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-a con 0x7fbb600a9750 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-b con 0x7fbb600add40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-c con 0x7fbb600aad20 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb60149870 con 0x7fbb600a9750 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb600abf80 con 0x7fbb600aad20 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb60149ac0 con 0x7fbb600add40 2024-03-29T11:32:50.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:50.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.788+0000 7fbb66634640 10 monclient(hunting): authenticate will time out at 827.026780s 2024-03-29T11:32:50.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.792+0000 7fbb65e33640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbb600aad20 0x7fbb60151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45195/0 (socket says 172.21.15.45:45195) 2024-03-29T11:32:50.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.792+0000 7fbb65e33640 1 -- 172.21.15.45:0/1297431673 learned_addr learned my addr 172.21.15.45:0/1297431673 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:50.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.794+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177862291 0 0) 0x7fbb600abf80 con 0x7fbb600aad20 2024-03-29T11:32:50.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4c003520 con 0x7fbb600aad20 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084390444 0 0) 0x7fbb60149870 con 0x7fbb600a9750 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:32:50.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb600abf80 con 0x7fbb600a9750 2024-03-29T11:32:50.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979708401 0 0) 0x7fbb600abf80 con 0x7fbb600a9750 2024-03-29T11:32:50.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 >> v1:172.21.15.45:6790/0 conn(0x7fbb600add40 legacy=0x7fbb60143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:50.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 >> v1:172.21.15.190:6789/0 conn(0x7fbb600aad20 legacy=0x7fbb60151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:50.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 monclient: found mon.noname-a 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6014a6f0 con 0x7fbb600a9750 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb66634640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb66634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbb5c001670 con 0x7fbb600a9750 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: dump: 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:50.793 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb66634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb5c001850 con 0x7fbb600a9750 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb66634640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.795+0000 7fbb66634640 4 set_mon_vals no callback set 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.796+0000 7fbb66634640 10 monclient: shutdown 2024-03-29T11:32:50.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.796+0000 7fbb66634640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.796+0000 7fbb66634640 1 -- 172.21.15.45:0/1297431673 >> v1:172.21.15.45:6789/0 conn(0x7fbb600a9750 legacy=0x7fbb60149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.796+0000 7fbb57fff640 1 -- 172.21.15.45:0/1297431673 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbb5c001670 con 0x7fbb600a9750 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.796+0000 7fbb57fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.798+0000 7fbb66634640 1 -- 172.21.15.45:0/1297431673 shutdown_connections 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.798+0000 7fbb66634640 1 -- 172.21.15.45:0/1297431673 >> 172.21.15.45:0/1297431673 conn(0x7fbb600673e0 msgr2=0x7fbb600b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:50.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.800+0000 7fbb66634640 1 -- 172.21.15.45:0/1297431673 shutdown_connections 2024-03-29T11:32:50.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.801+0000 7fbb66634640 1 -- 172.21.15.45:0/1297431673 wait complete. 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 10 monclient: build_initial_monmap 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 10 monclient: monmap: 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:50.797 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:50.798 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 librados: starting msgr at 2024-03-29T11:32:50.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 librados: starting objecter 2024-03-29T11:32:50.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 Processor -- start 2024-03-29T11:32:50.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 -- start start 2024-03-29T11:32:50.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 librados: setting wanted keys 2024-03-29T11:32:50.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 1 librados: calling monclient init 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.802+0000 7fbb66634640 10 monclient: init 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-a con 0x7fbb600a9750 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-c con 0x7fbb600aad20 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:50.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient(hunting): picked mon.noname-b con 0x7fbb600add40 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:50.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb601db3d0 con 0x7fbb600a9750 2024-03-29T11:32:50.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb60146aa0 con 0x7fbb600add40 2024-03-29T11:32:50.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6013ee50 con 0x7fbb600aad20 2024-03-29T11:32:50.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:50.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.803+0000 7fbb66634640 10 monclient(hunting): authenticate will time out at 827.041659s 2024-03-29T11:32:50.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.809+0000 7fbb64e31640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbb600aad20 0x7fbb601db860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60039/0 (socket says 172.21.15.45:60039) 2024-03-29T11:32:50.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.809+0000 7fbb64e31640 1 -- 172.21.15.45:0/78146994 learned_addr learned my addr 172.21.15.45:0/78146994 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:50.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.809+0000 7fbb55ffb640 10 client.?.objecter ms_handle_connect 0x7fbb600aad20 2024-03-29T11:32:50.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.809+0000 7fbb55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:50.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 client.?.objecter ms_handle_connect 0x7fbb600add40 2024-03-29T11:32:50.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028343526 0 0) 0x7fbb6013ee50 con 0x7fbb600aad20 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44004eb0 con 0x7fbb600aad20 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487895265 0 0) 0x7fbb60146aa0 con 0x7fbb600add40 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.810+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6013ee50 con 0x7fbb600add40 2024-03-29T11:32:50.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1743703843 0 0) 0x7fbb6013ee50 con 0x7fbb600add40 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb44005490 con 0x7fbb600add40 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbb500018d0 con 0x7fbb600add40 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:50.808 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.811+0000 7fbb55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245743531 0 0) 0x7fbb44004eb0 con 0x7fbb600aad20 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb44005b90 con 0x7fbb600aad20 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbb58001b70 con 0x7fbb600aad20 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.809 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 22055492 0 0) 0x7fbb44005b90 con 0x7fbb600aad20 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 >> v1:172.21.15.190:6789/0 conn(0x7fbb600add40 legacy=0x7fbb60071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:50.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.812+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 >> v1:172.21.15.45:6789/0 conn(0x7fbb600a9750 legacy=0x7fbb6006e060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:50.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:50.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 1 monclient: found mon.c 2024-03-29T11:32:50.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:50.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:50.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb601dbf70 con 0x7fbb600aad20 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 10 monclient: _renew_subs 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb601dc1c0 con 0x7fbb600aad20 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 10 monclient: _renew_subs 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6006f080 con 0x7fbb600aad20 2024-03-29T11:32:50.815 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.818+0000 7fbb66634640 1 librados: init done 2024-03-29T11:32:50.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.820+0000 7fbb66634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:50.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.820+0000 7fbb66634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:50.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.820+0000 7fbb66634640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:50.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.820+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6006f320 con 0x7fbb600aad20 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb58002060 con 0x7fbb600aad20 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbb58001b70 con 0x7fbb600aad20 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb55ffb640 10 monclient: dump: 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:50.817 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:50.818 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:50.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.821+0000 7fbb56ffd640 4 set_mon_vals no callback set 2024-03-29T11:32:50.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.826+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fbb5801aa70 con 0x7fbb600aad20 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fbb58084870 con 0x7fbb600aad20 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb55ffb640 10 client.4253.objecter ms_dispatch 0x7fbb60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb55ffb640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb55ffb640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb55ffb640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:32:50.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.827+0000 7fbb65632640 10 monclient: get_auth_request con 0x7fbb4406e790 auth_method 0 2024-03-29T11:32:50.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.828+0000 7fbb55ffb640 10 client.4253.objecter ms_handle_connect 0x7fbb4406e790 2024-03-29T11:32:50.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.832+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb5804ff80 con 0x7fbb600aad20 2024-03-29T11:32:50.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.832+0000 7fbb55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:50.827 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:50.832+0000 7fbb55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:51.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.151+0000 7fbb66634640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:51.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.151+0000 7fbb66634640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:51.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.151+0000 7fbb66634640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:32:51.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.151+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb60070300 con 0x7fbb600aad20 2024-03-29T11:32:51.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.152+0000 7fbb55ffb640 1 -- 172.21.15.45:0/78146994 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fbb58052080 con 0x7fbb600aad20 2024-03-29T11:32:51.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.152+0000 7fbb55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:51.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.152+0000 7fbb55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:51.147 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:51.147 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:51.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.156+0000 7fbb66634640 10 librados: watch_flush enter 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.159+0000 7fbb66634640 10 librados: watch_flush exit 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.159+0000 7fbb66634640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.159+0000 7fbb66634640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 >> v1:172.21.15.190:6816/1411598462 conn(0x7fbb4406e790 legacy=0x7fbb44070c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 10 monclient: shutdown 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 >> v1:172.21.15.45:6790/0 conn(0x7fbb600aad20 legacy=0x7fbb601db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 shutdown_connections 2024-03-29T11:32:51.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.160+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 >> 172.21.15.45:0/78146994 conn(0x7fbb600673e0 msgr2=0x7fbb60149fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:51.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.162+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 shutdown_connections 2024-03-29T11:32:51.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.162+0000 7fbb66634640 1 -- 172.21.15.45:0/78146994 wait complete. 2024-03-29T11:32:51.157 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:51.163+0000 7fbb66634640 1 librados: shutdown 2024-03-29T11:32:51.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:57.982 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:58.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.139+0000 7f032291c640 10 monclient: get_monmap_and_config 2024-03-29T11:32:58.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.139+0000 7f032291c640 10 monclient: build_initial_monmap 2024-03-29T11:32:58.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.139+0000 7f032291c640 10 monclient: monmap: 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:32:58.139589+0000 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:32:58.139589+0000 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:32:58.135 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.141+0000 7f032291c640 1 Processor -- start 2024-03-29T11:32:58.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.141+0000 7f032291c640 1 -- start start 2024-03-29T11:32:58.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.141+0000 7f032291c640 10 monclient: init 2024-03-29T11:32:58.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient(hunting): picked mon.noname-a con 0x7f031c0b0860 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient(hunting): picked mon.noname-c con 0x7f031c072450 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient(hunting): picked mon.noname-b con 0x7f031c13ee40 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0ad840 con 0x7f031c0b0860 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c13c830 con 0x7f031c072450 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c13c9c0 con 0x7f031c13ee40 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.142+0000 7f032291c640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:58.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.143+0000 7f032291c640 10 monclient(hunting): authenticate will time out at 834.381111s 2024-03-29T11:32:58.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.146+0000 7f0320eb9640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f031c13ee40 0x7f031c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50091/0 (socket says 172.21.15.45:50091) 2024-03-29T11:32:58.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.146+0000 7f0320eb9640 1 -- 172.21.15.45:0/1887648327 learned_addr learned my addr 172.21.15.45:0/1887648327 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396294718 0 0) 0x7f031c0ad840 con 0x7f031c0b0860 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304003520 con 0x7f031c0b0860 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220196339 0 0) 0x7f031c13c9c0 con 0x7f031c13ee40 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:32:58.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.148+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03040048a0 con 0x7f031c13ee40 2024-03-29T11:32:58.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473503951 0 0) 0x7f03040048a0 con 0x7f031c13ee40 2024-03-29T11:32:58.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 >> v1:172.21.15.190:6789/0 conn(0x7f031c072450 legacy=0x7f031c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 >> v1:172.21.15.45:6789/0 conn(0x7f031c0b0860 legacy=0x7f031c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 monclient: found mon.noname-b 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c13f970 con 0x7f031c13ee40 2024-03-29T11:32:58.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: _finish_auth 0 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0310001670 con 0x7f031c13ee40 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: dump: 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.149 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03100017d0 con 0x7f031c13ee40 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 1 -- 172.21.15.45:0/1887648327 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0310001be0 con 0x7f031c13ee40 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f031affd640 10 monclient: dump: 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:58.150 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f032291c640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f032291c640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.150+0000 7f032291c640 4 set_mon_vals no callback set 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.151+0000 7f032291c640 10 monclient: shutdown 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.151+0000 7f032291c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.151+0000 7f032291c640 1 -- 172.21.15.45:0/1887648327 >> v1:172.21.15.45:6790/0 conn(0x7f031c13ee40 legacy=0x7f031c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.152+0000 7f032291c640 1 -- 172.21.15.45:0/1887648327 shutdown_connections 2024-03-29T11:32:58.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.152+0000 7f032291c640 1 -- 172.21.15.45:0/1887648327 >> 172.21.15.45:0/1887648327 conn(0x7f031c06b660 msgr2=0x7f031c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:58.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.154+0000 7f032291c640 1 -- 172.21.15.45:0/1887648327 shutdown_connections 2024-03-29T11:32:58.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.157+0000 7f032291c640 1 -- 172.21.15.45:0/1887648327 wait complete. 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.158+0000 7f032291c640 10 monclient: build_initial_monmap 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.158+0000 7f032291c640 10 monclient: monmap: 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.153 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:32:58.154 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:32:58.154 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:32:58.154 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.158+0000 7f032291c640 1 librados: starting msgr at 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.158+0000 7f032291c640 1 librados: starting objecter 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.159+0000 7f032291c640 1 Processor -- start 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.159+0000 7f032291c640 1 -- start start 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.159+0000 7f032291c640 1 librados: setting wanted keys 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.159+0000 7f032291c640 1 librados: calling monclient init 2024-03-29T11:32:58.155 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.159+0000 7f032291c640 10 monclient: init 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient(hunting): picked mon.noname-c con 0x7f031c072450 addr v1:172.21.15.45:6790/0 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient(hunting): picked mon.noname-b con 0x7f031c0b0860 addr v1:172.21.15.190:6789/0 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient(hunting): picked mon.noname-a con 0x7f031c13ee40 addr v1:172.21.15.45:6789/0 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c12c570 con 0x7f031c13ee40 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c13d7a0 con 0x7f031c0b0860 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c13da80 con 0x7f031c072450 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:58.156 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.160+0000 7f032291c640 10 monclient(hunting): authenticate will time out at 834.398342s 2024-03-29T11:32:58.158 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.163+0000 7f0320eb9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f031c13ee40 0x7f031c12d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33023/0 (socket says 172.21.15.45:33023) 2024-03-29T11:32:58.158 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.164+0000 7f0320eb9640 1 -- 172.21.15.45:0/1118953721 learned_addr learned my addr 172.21.15.45:0/1118953721 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.164+0000 7f0318ff9640 10 client.?.objecter ms_handle_connect 0x7f031c13ee40 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186682698 0 0) 0x7f031c12c570 con 0x7f031c13ee40 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:58.160 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308004eb0 con 0x7f031c13ee40 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814275127 0 0) 0x7f0308004eb0 con 0x7f031c13ee40 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f031c12c570 con 0x7f031c13ee40 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0310001ba0 con 0x7f031c13ee40 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:58.161 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 client.?.objecter ms_handle_connect 0x7f031c0b0860 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.165+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1736918537 0 0) 0x7f031c12c570 con 0x7f031c13ee40 2024-03-29T11:32:58.162 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.166+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 >> v1:172.21.15.45:6790/0 conn(0x7f031c072450 legacy=0x7f031c13e4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.166+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 >> v1:172.21.15.190:6789/0 conn(0x7f031c0b0860 legacy=0x7f031c12ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.166+0000 7f0318ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.166+0000 7f0318ff9640 1 monclient: found mon.a 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c13dd60 con 0x7f031c13ee40 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:32:58.166 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 10 monclient: _renew_subs 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c13ec00 con 0x7f031c13ee40 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 10 monclient: _renew_subs 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c1e72c0 con 0x7f031c13ee40 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f032291c640 1 librados: init done 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.167+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0310004a90 con 0x7f031c13ee40 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.168+0000 7f0318ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.168+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0310004da0 con 0x7f031c13ee40 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.168+0000 7f0318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.168+0000 7f0318ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:58.167 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.168+0000 7f0318ff9640 10 monclient: dump: 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f031001ad70 con 0x7f031c13ee40 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f032291c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f032291c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f032291c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:58.168 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c0b0c40 con 0x7f031c13ee40 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f0310084bc0 con 0x7f031c13ee40 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 10 client.4269.objecter ms_dispatch 0x7f031c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.170+0000 7f0318ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.171+0000 7f031bfff640 10 monclient: get_auth_request con 0x7f030806dba0 auth_method 0 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.172+0000 7f0319ffb640 4 set_mon_vals no callback set 2024-03-29T11:32:58.169 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.172+0000 7f0318ff9640 10 client.4269.objecter ms_handle_connect 0x7f030806dba0 2024-03-29T11:32:58.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.179+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03100504a0 con 0x7f031c13ee40 2024-03-29T11:32:58.173 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.179+0000 7f0318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:58.174 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.179+0000 7f0318ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:58.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.505+0000 7f032291c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:58.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.506+0000 7f032291c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:58.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.506+0000 7f032291c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:32:58.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.506+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031c12c700 con 0x7f031c13ee40 2024-03-29T11:32:58.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.507+0000 7f0318ff9640 1 -- 172.21.15.45:0/1118953721 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f031c12c700 con 0x7f031c13ee40 2024-03-29T11:32:58.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.507+0000 7f0318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:58.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.507+0000 7f0318ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:58.503 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:32:58.503 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 10 librados: watch_flush enter 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 10 librados: watch_flush exit 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 >> v1:172.21.15.190:6816/1411598462 conn(0x7f030806dba0 legacy=0x7f0308070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 10 monclient: shutdown 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.511+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 >> v1:172.21.15.45:6789/0 conn(0x7f031c13ee40 legacy=0x7f031c12d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.512+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 shutdown_connections 2024-03-29T11:32:58.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.512+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 >> 172.21.15.45:0/1118953721 conn(0x7f031c06b660 msgr2=0x7f031c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:58.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.513+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 shutdown_connections 2024-03-29T11:32:58.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.513+0000 7f032291c640 1 -- 172.21.15.45:0/1118953721 wait complete. 2024-03-29T11:32:58.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:32:58.513+0000 7f032291c640 1 librados: shutdown 2024-03-29T11:32:58.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:05.332 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.499+0000 7f093ea96640 10 monclient: get_monmap_and_config 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.499+0000 7f093ea96640 10 monclient: build_initial_monmap 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.499+0000 7f093ea96640 10 monclient: monmap: 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:05.501096+0000 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:05.501096+0000 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.496 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:05.497 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:05.497 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:05.497 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.502+0000 7f093ea96640 1 Processor -- start 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.502+0000 7f093ea96640 1 -- start start 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.502+0000 7f093ea96640 10 monclient: init 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.503+0000 7f093ea96640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.503+0000 7f093ea96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.503+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-b con 0x7f09380b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:05.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.503+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-c con 0x7f0938071eb0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:05.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093d033640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0938143d10 0x7f0938146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40325/0 (socket says 172.21.15.45:40325) 2024-03-29T11:33:05.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093d033640 1 -- 172.21.15.45:0/1144174025 learned_addr learned my addr 172.21.15.45:0/1144174025 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:05.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-a con 0x7f0938143d10 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093806ef80 con 0x7f0938143d10 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093806f260 con 0x7f0938071eb0 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380b40c0 con 0x7f09380b1f60 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.506+0000 7f093ea96640 10 monclient(hunting): authenticate will time out at 841.745641s 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.507+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192609326 0 0) 0x7f093806ef80 con 0x7f0938143d10 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920003520 con 0x7f0938143d10 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796532591 0 0) 0x7f09380b40c0 con 0x7f09380b1f60 2024-03-29T11:33:05.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093806ef80 con 0x7f09380b1f60 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237491761 0 0) 0x7f0920003520 con 0x7f0938143d10 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 >> v1:172.21.15.45:6790/0 conn(0x7f09380b1f60 legacy=0x7f0938143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.508+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 >> v1:172.21.15.190:6789/0 conn(0x7f0938071eb0 legacy=0x7f0938146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:05.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:05.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 1 monclient: found mon.noname-a 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938154d60 con 0x7f0938143d10 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f093ea96640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f093ea96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f092c001690 con 0x7f0938143d10 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.510+0000 7f09377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f09377fe640 10 monclient: dump: 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:05.511 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f093ea96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f092c0018b0 con 0x7f0938143d10 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f09377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f093ea96640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f093ea96640 4 set_mon_vals no callback set 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f09377fe640 1 -- 172.21.15.45:0/1144174025 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f092c001c40 con 0x7f0938143d10 2024-03-29T11:33:05.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.511+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f09377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f09377fe640 10 monclient: dump: 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 10 monclient: shutdown 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:05.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 >> v1:172.21.15.45:6789/0 conn(0x7f0938143d10 legacy=0x7f0938146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 shutdown_connections 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 >> 172.21.15.45:0/1144174025 conn(0x7f0938066a00 msgr2=0x7f0938066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 shutdown_connections 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.512+0000 7f093ea96640 1 -- 172.21.15.45:0/1144174025 wait complete. 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.513+0000 7f093ea96640 10 monclient: build_initial_monmap 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.513+0000 7f093ea96640 10 monclient: monmap: 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:05.514 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.513+0000 7f093ea96640 1 librados: starting msgr at 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.513+0000 7f093ea96640 1 librados: starting objecter 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.514+0000 7f093ea96640 1 Processor -- start 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.514+0000 7f093ea96640 1 -- start start 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.514+0000 7f093ea96640 1 librados: setting wanted keys 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.514+0000 7f093ea96640 1 librados: calling monclient init 2024-03-29T11:33:05.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.514+0000 7f093ea96640 10 monclient: init 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.515+0000 7f093ea96640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.516+0000 7f093ea96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.516+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-b con 0x7f09380b1f60 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.516+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-a con 0x7f0938143d10 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.519+0000 7f093d033640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f09380b6f50 0x7f0938140010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47179/0 (socket says 172.21.15.45:47179) 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.519+0000 7f093d033640 1 -- 172.21.15.45:0/4261560646 learned_addr learned my addr 172.21.15.45:0/4261560646 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.519+0000 7f093ea96640 10 monclient(hunting): picked mon.noname-c con 0x7f09380b6f50 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.519+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380ac030 con 0x7f0938143d10 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0938140720 con 0x7f09380b1f60 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380ad630 con 0x7f09380b6f50 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f093ea96640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f093ea96640 10 monclient(hunting): authenticate will time out at 841.759471s 2024-03-29T11:33:05.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 client.?.objecter ms_handle_connect 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134309724 0 0) 0x7f09380ad630 con 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.520+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0924004eb0 con 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.521+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897309357 0 0) 0x7f0924004eb0 con 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.521+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09380ad630 con 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.521+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f092c001bc0 con 0x7f09380b6f50 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.521+0000 7f09357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:05.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.521+0000 7f09357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:05.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.523+0000 7f09357fa640 10 monclient(hunting): dump: 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.523+0000 7f09357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.523+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 698996251 0 0) 0x7f09380ad630 con 0x7f09380b6f50 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.523+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 >> v1:172.21.15.190:6789/0 conn(0x7f09380b1f60 legacy=0x7f09381e3e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:33:05.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.523+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 >> v1:172.21.15.45:6789/0 conn(0x7f0938143d10 legacy=0x7f093813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 1 monclient: found mon.c 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380aafd0 con 0x7f09380b6f50 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 client.?.objecter ms_handle_connect 0x7f0938143d10 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f092c004b00 con 0x7f09380b6f50 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f092c004f50 con 0x7f09380b6f50 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:05.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09357fa640 10 monclient: dump: 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f09367fc640 4 set_mon_vals no callback set 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f093ea96640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:33:05.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f093ea96640 10 monclient: _renew_subs 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f093ea96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381e4520 con 0x7f09380b6f50 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.524+0000 7f093ea96640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.525+0000 7f093ea96640 10 monclient: _renew_subs 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.525+0000 7f093ea96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.525+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09380ab840 con 0x7f09380b6f50 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.525+0000 7f093ea96640 1 librados: init done 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.526+0000 7f093ea96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.526+0000 7f093ea96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.526+0000 7f093ea96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:05.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.526+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0938075f10 con 0x7f09380b6f50 2024-03-29T11:33:05.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.529+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f092c01af20 con 0x7f09380b6f50 2024-03-29T11:33:05.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f092c084b60 con 0x7f09380b6f50 2024-03-29T11:33:05.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f093c832640 10 monclient: get_auth_request con 0x7f092406db80 auth_method 0 2024-03-29T11:33:05.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f09357fa640 10 client.4262.objecter ms_dispatch 0x7f0938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:05.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f09357fa640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:05.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f09357fa640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:05.528 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.530+0000 7f09357fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:33:05.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.534+0000 7f09357fa640 10 client.4262.objecter ms_handle_connect 0x7f092406db80 2024-03-29T11:33:05.533 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.537+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f092c050440 con 0x7f09380b6f50 2024-03-29T11:33:05.533 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.538+0000 7f09357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:05.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.538+0000 7f09357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:05.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.870+0000 7f093ea96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:05.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.870+0000 7f093ea96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:05.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.870+0000 7f093ea96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:05.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.870+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09380ac1e0 con 0x7f09380b6f50 2024-03-29T11:33:05.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.871+0000 7f09357fa640 1 -- 172.21.15.45:0/4261560646 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f09380ac1e0 con 0x7f09380b6f50 2024-03-29T11:33:05.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.871+0000 7f09357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:05.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.871+0000 7f09357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:05.870 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:05.871 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:05.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.877+0000 7f093ea96640 10 librados: watch_flush enter 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 10 librados: watch_flush exit 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 >> v1:172.21.15.190:6816/1411598462 conn(0x7f092406db80 legacy=0x7f0924070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 10 monclient: shutdown 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:05.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 >> v1:172.21.15.45:6790/0 conn(0x7f09380b6f50 legacy=0x7f0938140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:05.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 shutdown_connections 2024-03-29T11:33:05.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.878+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 >> 172.21.15.45:0/4261560646 conn(0x7f0938066a00 msgr2=0x7f0938146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:05.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.879+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 shutdown_connections 2024-03-29T11:33:05.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.879+0000 7f093ea96640 1 -- 172.21.15.45:0/4261560646 wait complete. 2024-03-29T11:33:05.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:05.879+0000 7f093ea96640 1 librados: shutdown 2024-03-29T11:33:05.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:12.703 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:12.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.861+0000 7fa4b4d05640 10 monclient: get_monmap_and_config 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.861+0000 7fa4b4d05640 10 monclient: build_initial_monmap 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.861+0000 7fa4b4d05640 10 monclient: monmap: 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:12.862831+0000 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:12.862831+0000 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:12.859 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.863+0000 7fa4b4d05640 1 Processor -- start 2024-03-29T11:33:12.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.864+0000 7fa4b4d05640 1 -- start start 2024-03-29T11:33:12.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.864+0000 7fa4b4d05640 10 monclient: init 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.864+0000 7fa4b4d05640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b00a9750 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b00ae220 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b013b9d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0142490 con 0x7fa4b00a9750 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00ac130 con 0x7fa4b00ae220 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00ac410 con 0x7fa4b013b9d0 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:12.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.865+0000 7fa4b4d05640 10 monclient(hunting): authenticate will time out at 849.104206s 2024-03-29T11:33:12.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.868+0000 7fa4affff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa4b013b9d0 0x7fa4b0149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47453/0 (socket says 172.21.15.45:47453) 2024-03-29T11:33:12.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.868+0000 7fa4affff640 1 -- 172.21.15.45:0/1292773418 learned_addr learned my addr 172.21.15.45:0/1292773418 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391689970 0 0) 0x7fa4b00ac410 con 0x7fa4b013b9d0 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498005730 con 0x7fa4b013b9d0 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2530834063 0 0) 0x7fa498005730 con 0x7fa4b013b9d0 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 >> v1:172.21.15.190:6789/0 conn(0x7fa4b00ae220 legacy=0x7fa4b013b2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:33:12.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.869+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 >> v1:172.21.15.45:6789/0 conn(0x7fa4b00a9750 legacy=0x7fa4b0141d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:12.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 1 monclient: found mon.noname-b 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00afa40 con 0x7fa4b013b9d0 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa4a4001690 con 0x7fa4b013b9d0 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4b4d05640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: dump: 2024-03-29T11:33:12.871 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4b4d05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4a40017f0 con 0x7fa4b013b9d0 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4b4d05640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:12.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.874+0000 7fa4b4d05640 4 set_mon_vals no callback set 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4ae7fc640 1 -- 172.21.15.45:0/1292773418 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa4a4001c40 con 0x7fa4b013b9d0 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4ae7fc640 10 monclient: dump: 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.873 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4b4d05640 10 monclient: shutdown 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4b4d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4b4d05640 1 -- 172.21.15.45:0/1292773418 >> v1:172.21.15.45:6790/0 conn(0x7fa4b013b9d0 legacy=0x7fa4b0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4b4d05640 1 -- 172.21.15.45:0/1292773418 shutdown_connections 2024-03-29T11:33:12.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.875+0000 7fa4b4d05640 1 -- 172.21.15.45:0/1292773418 >> 172.21.15.45:0/1292773418 conn(0x7fa4b00673e0 msgr2=0x7fa4b00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:12.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.878+0000 7fa4b4d05640 1 -- 172.21.15.45:0/1292773418 shutdown_connections 2024-03-29T11:33:12.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.878+0000 7fa4b4d05640 1 -- 172.21.15.45:0/1292773418 wait complete. 2024-03-29T11:33:12.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.879+0000 7fa4b4d05640 10 monclient: build_initial_monmap 2024-03-29T11:33:12.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.879+0000 7fa4b4d05640 10 monclient: monmap: 2024-03-29T11:33:12.880 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.880+0000 7fa4b4d05640 1 librados: starting msgr at 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.880+0000 7fa4b4d05640 1 librados: starting objecter 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.880+0000 7fa4b4d05640 1 Processor -- start 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.882+0000 7fa4b4d05640 1 -- start start 2024-03-29T11:33:12.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.882+0000 7fa4b4d05640 1 librados: setting wanted keys 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.882+0000 7fa4b4d05640 1 librados: calling monclient init 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.882+0000 7fa4b4d05640 10 monclient: init 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b00a9750 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b00ae220 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b013b9d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b006fd60 con 0x7fa4b00a9750 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0070040 con 0x7fa4b00ae220 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00b2c80 con 0x7fa4b013b9d0 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:12.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.883+0000 7fa4b4d05640 10 monclient(hunting): authenticate will time out at 849.122332s 2024-03-29T11:33:12.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.887+0000 7fa4aeffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa4b00ae220 0x7fa4b00b2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:43973/0 (socket says 172.21.15.45:43973) 2024-03-29T11:33:12.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.887+0000 7fa4aeffd640 1 -- 172.21.15.45:0/386708354 learned_addr learned my addr 172.21.15.45:0/386708354 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:12.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 client.?.objecter ms_handle_connect 0x7fa4b013b9d0 2024-03-29T11:33:12.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:12.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34943688 0 0) 0x7fa4b00b2c80 con 0x7fa4b013b9d0 2024-03-29T11:33:12.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c004eb0 con 0x7fa4b013b9d0 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.888+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2503188053 0 0) 0x7fa49c004eb0 con 0x7fa4b013b9d0 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4b00b2c80 con 0x7fa4b013b9d0 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa4a4001b00 con 0x7fa4b013b9d0 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:12.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient(hunting): dump: 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 422914153 0 0) 0x7fa4b00b2c80 con 0x7fa4b013b9d0 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 >> v1:172.21.15.190:6789/0 conn(0x7fa4b00ae220 legacy=0x7fa4b00b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 >> v1:172.21.15.45:6789/0 conn(0x7fa4b00a9750 legacy=0x7fa4b0071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 monclient: found mon.c 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0070460 con 0x7fa4b013b9d0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 10 monclient: _renew_subs 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b006ec70 con 0x7fa4b013b9d0 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:12.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 10 monclient: _renew_subs 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0072610 con 0x7fa4b013b9d0 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.889+0000 7fa4b4d05640 1 librados: init done 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.890+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4a4001c60 con 0x7fa4b013b9d0 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.890+0000 7fa497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.890+0000 7fa4adffb640 4 set_mon_vals no callback set 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.891+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa4a4004c30 con 0x7fa4b013b9d0 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.891+0000 7fa497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.891+0000 7fa497fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.891+0000 7fa497fff640 10 monclient: dump: 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.896 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fa4a401ac00 con 0x7fa4b013b9d0 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fa4a4084bb0 con 0x7fa4b013b9d0 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 10 client.4268.objecter ms_dispatch 0x7fa4b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.894+0000 7fa497fff640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa4b4d05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa4b4d05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:12.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa4b4d05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:12.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa490005080 con 0x7fa4b013b9d0 2024-03-29T11:33:12.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa4af7fe640 10 monclient: get_auth_request con 0x7fa49c06dcd0 auth_method 0 2024-03-29T11:33:12.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.895+0000 7fa497fff640 10 client.4268.objecter ms_handle_connect 0x7fa49c06dcd0 2024-03-29T11:33:12.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.906+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4a40502c0 con 0x7fa4b013b9d0 2024-03-29T11:33:12.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.906+0000 7fa497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:12.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:12.906+0000 7fa497fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:13.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.232+0000 7fa4b4d05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:13.231 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:13.231 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.232+0000 7fa4b4d05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.232+0000 7fa4b4d05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.232+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa490005330 con 0x7fa4b013b9d0 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.232+0000 7fa497fff640 1 -- 172.21.15.45:0/386708354 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fa4a40523c0 con 0x7fa4b013b9d0 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.233+0000 7fa497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.233+0000 7fa497fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 10 librados: watch_flush enter 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 10 librados: watch_flush exit 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 10 client.4268.objecter successfully canceled tick 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 >> v1:172.21.15.190:6816/1411598462 conn(0x7fa49c06dcd0 legacy=0x7fa49c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 10 monclient: shutdown 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 >> v1:172.21.15.45:6790/0 conn(0x7fa4b013b9d0 legacy=0x7fa4b0070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 shutdown_connections 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.238+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 >> 172.21.15.45:0/386708354 conn(0x7fa4b00673e0 msgr2=0x7fa4b0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.239+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 shutdown_connections 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.239+0000 7fa4b4d05640 1 -- 172.21.15.45:0/386708354 wait complete. 2024-03-29T11:33:13.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:13.239+0000 7fa4b4d05640 1 librados: shutdown 2024-03-29T11:33:13.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:20.078 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.234+0000 7f66ddadb640 10 monclient: get_monmap_and_config 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.234+0000 7f66ddadb640 10 monclient: build_initial_monmap 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.234+0000 7f66ddadb640 10 monclient: monmap: 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:20.236090+0000 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:20.236090+0000 2024-03-29T11:33:20.231 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:20.232 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.232 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:20.232 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:20.232 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:20.232 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.235+0000 7f66ddadb640 1 Processor -- start 2024-03-29T11:33:20.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 1 -- start start 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient: init 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-c con 0x7f66d00389e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-a con 0x7f66d0038dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-b con 0x7f66d0030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00c54c0 con 0x7f66d0038dc0 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d0036430 con 0x7f66d00389e0 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d0039840 con 0x7f66d0030660 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:20.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.236+0000 7f66ddadb640 10 monclient(hunting): authenticate will time out at 856.476069s 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66dd2da640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f66d0030660 0x7f66d00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51842/0 (socket says 172.21.15.45:51842) 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f66d0038dc0 0x7f66d00cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:39387/0 (socket says 172.21.15.45:39387) 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66dd2da640 1 -- 172.21.15.45:0/3339514442 learned_addr learned my addr 172.21.15.45:0/3339514442 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305434816 0 0) 0x7f66d0039840 con 0x7f66d0030660 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c8003520 con 0x7f66d0030660 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042296658 0 0) 0x7f66d00c54c0 con 0x7f66d0038dc0 2024-03-29T11:33:20.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.240+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d0039840 con 0x7f66d0038dc0 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700373008 0 0) 0x7f66c8003520 con 0x7f66d0030660 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 >> v1:172.21.15.190:6789/0 conn(0x7f66d00389e0 legacy=0x7f66d0033520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 >> v1:172.21.15.45:6789/0 conn(0x7f66d0038dc0 legacy=0x7f66d00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 monclient: found mon.noname-b 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d0035690 con 0x7f66d0030660 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:20.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f66d804f730 con 0x7f66d0030660 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66d77fe640 10 monclient: dump: 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:20.243 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66ddadb640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.241+0000 7f66ddadb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66d80008d0 con 0x7f66d0030660 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66ddadb640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66ddadb640 4 set_mon_vals no callback set 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 1 -- 172.21.15.45:0/3339514442 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f66d80758b0 con 0x7f66d0030660 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.242+0000 7f66d77fe640 10 monclient: dump: 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.244 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 10 monclient: shutdown 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 1 -- 172.21.15.45:0/3339514442 >> v1:172.21.15.45:6790/0 conn(0x7f66d0030660 legacy=0x7f66d00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 1 -- 172.21.15.45:0/3339514442 shutdown_connections 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 1 -- 172.21.15.45:0/3339514442 >> 172.21.15.45:0/3339514442 conn(0x7f66d0005c00 msgr2=0x7f66d0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 1 -- 172.21.15.45:0/3339514442 shutdown_connections 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.243+0000 7f66ddadb640 1 -- 172.21.15.45:0/3339514442 wait complete. 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.244+0000 7f66ddadb640 10 monclient: build_initial_monmap 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.245+0000 7f66ddadb640 10 monclient: monmap: 2024-03-29T11:33:20.245 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:20.246 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.245+0000 7f66ddadb640 1 librados: starting msgr at 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.245+0000 7f66ddadb640 1 librados: starting objecter 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.245+0000 7f66ddadb640 1 Processor -- start 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.246+0000 7f66ddadb640 1 -- start start 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.246+0000 7f66ddadb640 1 librados: setting wanted keys 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.246+0000 7f66ddadb640 1 librados: calling monclient init 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.246+0000 7f66ddadb640 10 monclient: init 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.247+0000 7f66ddadb640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.247+0000 7f66ddadb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:20.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.247+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-c con 0x7f66d0030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.247+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-b con 0x7f66d00389e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 10 monclient(hunting): picked mon.noname-a con 0x7f66d00c7960 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00c8480 con 0x7f66d00c7960 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00c9ae0 con 0x7f66d00389e0 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00c7e40 con 0x7f66d0030660 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66ddadb640 10 monclient(hunting): authenticate will time out at 856.487689s 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.248+0000 7f66dcad9640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f66d0030660 0x7f66d00c0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52834/0 (socket says 172.21.15.45:52834) 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.251+0000 7f66dcad9640 1 -- 172.21.15.45:0/1836340624 learned_addr learned my addr 172.21.15.45:0/1836340624 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.251+0000 7f66d57fa640 10 client.?.objecter ms_handle_connect 0x7f66d0030660 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.251+0000 7f66d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.252+0000 7f66d57fa640 10 client.?.objecter ms_handle_connect 0x7f66d00c7960 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.252+0000 7f66d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.252+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282273006 0 0) 0x7f66d00c7e40 con 0x7f66d0030660 2024-03-29T11:33:20.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.252+0000 7f66d57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66bc004eb0 con 0x7f66d0030660 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183065167 0 0) 0x7f66d00c8480 con 0x7f66d00c7960 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00c7e40 con 0x7f66d00c7960 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 client.?.objecter ms_handle_connect 0x7f66d00389e0 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722194629 0 0) 0x7f66bc004eb0 con 0x7f66d0030660 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.253+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d00c8480 con 0x7f66d0030660 2024-03-29T11:33:20.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f66cc001870 con 0x7f66d0030660 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): dump: 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:20.254 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:20.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791541896 0 0) 0x7f66d00c7e40 con 0x7f66d00c7960 2024-03-29T11:33:20.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66bc004eb0 con 0x7f66d00c7960 2024-03-29T11:33:20.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f66d8068810 con 0x7f66d00c7960 2024-03-29T11:33:20.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:20.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:20.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient(hunting): dump: 2024-03-29T11:33:20.256 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:20.256 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037565196 0 0) 0x7f66d00c8480 con 0x7f66d0030660 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 >> v1:172.21.15.190:6789/0 conn(0x7f66d00389e0 legacy=0x7f66d00c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 >> v1:172.21.15.45:6789/0 conn(0x7f66d00c7960 legacy=0x7f66d00ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 monclient: found mon.c 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:20.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.254+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00c7fd0 con 0x7f66d0030660 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 10 monclient: _renew_subs 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d00c0b60 con 0x7f66d0030660 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66cc001e00 con 0x7f66d0030660 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 10 monclient: _renew_subs 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d002f910 con 0x7f66d0030660 2024-03-29T11:33:20.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66ddadb640 1 librados: init done 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f66cc004aa0 con 0x7f66d0030660 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.255+0000 7f66d57fa640 10 monclient: dump: 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:20.259 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.256+0000 7f66d6ffd640 4 set_mon_vals no callback set 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f66cc01aa70 con 0x7f66d0030660 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66ddadb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66ddadb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66ddadb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0007b70 con 0x7f66d0030660 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.258+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f66cc084a30 con 0x7f66d0030660 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d57fa640 10 client.4274.objecter ms_dispatch 0x7f66d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d7fff640 10 monclient: get_auth_request con 0x7f66bc06dca0 auth_method 0 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d57fa640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d57fa640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d57fa640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:33:20.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.259+0000 7f66d57fa640 10 client.4274.objecter ms_handle_connect 0x7f66bc06dca0 2024-03-29T11:33:20.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.269+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66cc050310 con 0x7f66d0030660 2024-03-29T11:33:20.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.269+0000 7f66d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:20.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.269+0000 7f66d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:20.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.606+0000 7f66ddadb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:20.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.606+0000 7f66ddadb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:20.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.606+0000 7f66ddadb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:20.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.606+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66d00c88f0 con 0x7f66d0030660 2024-03-29T11:33:20.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.607+0000 7f66d57fa640 1 -- 172.21.15.45:0/1836340624 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f66d00c88f0 con 0x7f66d0030660 2024-03-29T11:33:20.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.607+0000 7f66d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:20.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.608+0000 7f66d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:20.604 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:20.604 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:20.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.611+0000 7f66ddadb640 10 librados: watch_flush enter 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.611+0000 7f66ddadb640 10 librados: watch_flush exit 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.611+0000 7f66ddadb640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.611+0000 7f66ddadb640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 >> v1:172.21.15.190:6816/1411598462 conn(0x7f66bc06dca0 legacy=0x7f66bc070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 10 monclient: shutdown 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:20.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 >> v1:172.21.15.45:6790/0 conn(0x7f66d0030660 legacy=0x7f66d00c0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:20.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 shutdown_connections 2024-03-29T11:33:20.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 >> 172.21.15.45:0/1836340624 conn(0x7f66d0005c00 msgr2=0x7f66d0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:20.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 shutdown_connections 2024-03-29T11:33:20.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 -- 172.21.15.45:0/1836340624 wait complete. 2024-03-29T11:33:20.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:20.612+0000 7f66ddadb640 1 librados: shutdown 2024-03-29T11:33:20.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:27.431 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.593+0000 7f71e1a57640 10 monclient: get_monmap_and_config 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.593+0000 7f71e1a57640 10 monclient: build_initial_monmap 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.593+0000 7f71e1a57640 10 monclient: monmap: 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:27.594477+0000 2024-03-29T11:33:27.590 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:27.594477+0000 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:27.591 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.594+0000 7f71e1a57640 1 Processor -- start 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.594+0000 7f71e1a57640 1 -- start start 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.594+0000 7f71e1a57640 10 monclient: init 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.595+0000 7f71e1a57640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.595+0000 7f71e1a57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:27.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:27.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc13f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:27.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71daffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f71dc1457f0 0x7f71dc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59322/0 (socket says 172.21.15.45:59322) 2024-03-29T11:33:27.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-a con 0x7f71dc0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:27.593 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc0aaef0 con 0x7f71dc0a9d70 2024-03-29T11:33:27.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc145bd0 con 0x7f71dc13f650 2024-03-29T11:33:27.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc145eb0 con 0x7f71dc1457f0 2024-03-29T11:33:27.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71e1a57640 10 monclient(hunting): authenticate will time out at 863.835533s 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71d27fc640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f71dc13f650 0x7f71dc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:43264/0 (socket says 172.21.15.45:43264) 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.596+0000 7f71d27fc640 1 -- 172.21.15.45:0/2853088100 learned_addr learned my addr 172.21.15.45:0/2853088100 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44405023 0 0) 0x7f71dc145eb0 con 0x7f71dc1457f0 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c8003520 con 0x7f71dc1457f0 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886067181 0 0) 0x7f71dc145bd0 con 0x7f71dc13f650 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:27.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.597+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c8003840 con 0x7f71dc13f650 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772302327 0 0) 0x7f71c8003840 con 0x7f71dc13f650 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 >> v1:172.21.15.45:6790/0 conn(0x7f71dc1457f0 legacy=0x7f71dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 >> v1:172.21.15.45:6789/0 conn(0x7f71dc0a9d70 legacy=0x7f71dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 monclient: found mon.noname-c 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc146340 con 0x7f71dc13f650 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:27.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71e1a57640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71e1a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f71c4001630 con 0x7f71dc13f650 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: dump: 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.604 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71e1a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71c4001880 con 0x7f71dc13f650 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.598+0000 7f71da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71da7fc640 1 -- 172.21.15.45:0/2853088100 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f71c4001c10 con 0x7f71dc13f650 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71da7fc640 10 monclient: dump: 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:27.605 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71e1a57640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71e1a57640 4 set_mon_vals no callback set 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.599+0000 7f71e1a57640 10 monclient: shutdown 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 1 -- 172.21.15.45:0/2853088100 >> v1:172.21.15.190:6789/0 conn(0x7f71dc13f650 legacy=0x7f71dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 1 -- 172.21.15.45:0/2853088100 shutdown_connections 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 1 -- 172.21.15.45:0/2853088100 >> 172.21.15.45:0/2853088100 conn(0x7f71dc066a00 msgr2=0x7f71dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 1 -- 172.21.15.45:0/2853088100 shutdown_connections 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.600+0000 7f71e1a57640 1 -- 172.21.15.45:0/2853088100 wait complete. 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.601+0000 7f71e1a57640 10 monclient: build_initial_monmap 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.601+0000 7f71e1a57640 10 monclient: monmap: 2024-03-29T11:33:27.606 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.601+0000 7f71e1a57640 1 librados: starting msgr at 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 1 librados: starting objecter 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 1 Processor -- start 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 1 -- start start 2024-03-29T11:33:27.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 1 librados: setting wanted keys 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 1 librados: calling monclient init 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.602+0000 7f71e1a57640 10 monclient: init 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-a con 0x7f71dc0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc140830 con 0x7f71dc0a9d70 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71e1a57640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc13b390 con 0x7f71dc1457f0 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71db7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f71dc1457f0 0x7f71dc1d7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:43598/0 (socket says 172.21.15.45:43598) 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.603+0000 7f71db7fe640 1 -- 172.21.15.45:0/1588400539 learned_addr learned my addr 172.21.15.45:0/1588400539 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc13ba20 con 0x7f71dc13f650 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71e1a57640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:27.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71e1a57640 10 monclient(hunting): authenticate will time out at 863.843260s 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 client.?.objecter ms_handle_connect 0x7f71dc1457f0 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 client.?.objecter ms_handle_connect 0x7f71dc0a9d70 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789135708 0 0) 0x7f71dc13b390 con 0x7f71dc1457f0 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc004eb0 con 0x7f71dc1457f0 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868490722 0 0) 0x7f71dc140830 con 0x7f71dc0a9d70 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:27.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:33:27.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.604+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc13b390 con 0x7f71dc0a9d70 2024-03-29T11:33:27.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620491600 0 0) 0x7f71bc004eb0 con 0x7f71dc1457f0 2024-03-29T11:33:27.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71dc140830 con 0x7f71dc1457f0 2024-03-29T11:33:27.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f71d40019b0 con 0x7f71dc1457f0 2024-03-29T11:33:27.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:27.614 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 client.?.objecter ms_handle_connect 0x7f71dc13f650 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1192722925 0 0) 0x7f71dc13b390 con 0x7f71dc0a9d70 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71bc004eb0 con 0x7f71dc0a9d70 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f71cc001b90 con 0x7f71dc0a9d70 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:27.615 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1022797358 0 0) 0x7f71dc140830 con 0x7f71dc1457f0 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 >> v1:172.21.15.45:6790/0 conn(0x7f71dc13f650 legacy=0x7f71dc1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 >> v1:172.21.15.45:6789/0 conn(0x7f71dc0a9d70 legacy=0x7f71dc139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 monclient: found mon.b 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.605+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc13bbb0 con 0x7f71dc1457f0 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:33:27.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 10 monclient: _renew_subs 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc138fc0 con 0x7f71dc1457f0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 10 monclient: _renew_subs 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc1359c0 con 0x7f71dc1457f0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71e1a57640 1 librados: init done 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d4001b10 con 0x7f71dc1457f0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.606+0000 7f71d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.607+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f71d4003a60 con 0x7f71dc1457f0 2024-03-29T11:33:27.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.607+0000 7f71d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.607+0000 7f71d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.607+0000 7f71d8ff9640 10 monclient: dump: 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:27.618 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.609+0000 7f71e1a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.609+0000 7f71e1a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.609+0000 7f71e1a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.609+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc075f10 con 0x7f71dc1457f0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.609+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f71d4003bc0 con 0x7f71dc1457f0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f71d4084ac0 con 0x7f71dc1457f0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71d8ff9640 10 client.4261.objecter ms_dispatch 0x7f71dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71d8ff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71d8ff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71d8ff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.610+0000 7f71daffd640 10 monclient: get_auth_request con 0x7f71bc06de90 auth_method 0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.611+0000 7f71d8ff9640 10 client.4261.objecter ms_handle_connect 0x7f71bc06de90 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.617+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71d40503a0 con 0x7f71dc1457f0 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.617+0000 7f71d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:27.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:27.617+0000 7f71d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.020+0000 7f71e1a57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.020+0000 7f71e1a57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.020+0000 7f71e1a57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.020+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71dc139250 con 0x7f71dc1457f0 2024-03-29T11:33:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.021+0000 7f71d8ff9640 1 -- 172.21.15.45:0/1588400539 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f71d40522d0 con 0x7f71dc1457f0 2024-03-29T11:33:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.021+0000 7f71d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.021+0000 7f71d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:28.018 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:28.018 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.026+0000 7f71e1a57640 10 librados: watch_flush enter 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.026+0000 7f71e1a57640 10 librados: watch_flush exit 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.026+0000 7f71e1a57640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.026+0000 7f71e1a57640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.026+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 >> v1:172.21.15.190:6816/1411598462 conn(0x7f71bc06de90 legacy=0x7f71bc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.027+0000 7f71e1a57640 10 monclient: shutdown 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.027+0000 7f71e1a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.027+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 >> v1:172.21.15.190:6789/0 conn(0x7f71dc1457f0 legacy=0x7f71dc1d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.028+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 shutdown_connections 2024-03-29T11:33:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.028+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 >> 172.21.15.45:0/1588400539 conn(0x7f71dc066a00 msgr2=0x7f71dc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.028+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 shutdown_connections 2024-03-29T11:33:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.028+0000 7f71e1a57640 1 -- 172.21.15.45:0/1588400539 wait complete. 2024-03-29T11:33:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:28.028+0000 7f71e1a57640 1 librados: shutdown 2024-03-29T11:33:28.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:34.850 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:34.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.990+0000 7f4c291fe640 10 monclient: get_monmap_and_config 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.990+0000 7f4c291fe640 10 monclient: build_initial_monmap 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.990+0000 7f4c291fe640 10 monclient: monmap: 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:34.991421+0000 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:34.991421+0000 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:34.987 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:34.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.991+0000 7f4c291fe640 1 Processor -- start 2024-03-29T11:33:34.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.991+0000 7f4c291fe640 1 -- start start 2024-03-29T11:33:34.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.991+0000 7f4c291fe640 10 monclient: init 2024-03-29T11:33:34.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:34.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:33:34.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2406c250 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:34.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4c240ae650 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:34.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2413f8c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240adf50 con 0x7f4c240ae650 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c2359e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4c2413f8c0 0x7f4c2414db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:57956/0 (socket says 172.21.15.45:57956) 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c2359e640 1 -- 172.21.15.45:0/2122771864 learned_addr learned my addr 172.21.15.45:0/2122771864 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240ae230 con 0x7f4c2413f8c0 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2413fd50 con 0x7f4c2406c250 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:34.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.992+0000 7f4c291fe640 10 monclient(hunting): authenticate will time out at 871.232016s 2024-03-29T11:33:34.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291370343 0 0) 0x7f4c2413fd50 con 0x7f4c2406c250 2024-03-29T11:33:34.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:34.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c003520 con 0x7f4c2406c250 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155939076 0 0) 0x7f4c240ae230 con 0x7f4c2413f8c0 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.993+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2413fd50 con 0x7f4c2413f8c0 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.994+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321570479 0 0) 0x7f4c0c003520 con 0x7f4c2406c250 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.994+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 >> v1:172.21.15.190:6789/0 conn(0x7f4c2413f8c0 legacy=0x7f4c2414db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.994+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 >> v1:172.21.15.45:6789/0 conn(0x7f4c240ae650 legacy=0x7f4c2413f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:33:34.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 1 monclient: found mon.noname-b 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2406db20 con 0x7f4c2406c250 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: _finish_auth 0 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4c18001690 con 0x7f4c2406c250 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: dump: 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:34.995 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c291fe640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c291fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c180018b0 con 0x7f4c2406c250 2024-03-29T11:33:34.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c291fe640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.995+0000 7f4c291fe640 4 set_mon_vals no callback set 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c21d9b640 1 -- 172.21.15.45:0/2122771864 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4c18001c40 con 0x7f4c2406c250 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c21d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c21d9b640 10 monclient: dump: 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:34.997 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 10 monclient: shutdown 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 >> v1:172.21.15.45:6790/0 conn(0x7f4c2406c250 legacy=0x7f4c241427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 shutdown_connections 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 >> 172.21.15.45:0/2122771864 conn(0x7f4c24067610 msgr2=0x7f4c240b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 shutdown_connections 2024-03-29T11:33:34.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.996+0000 7f4c291fe640 1 -- 172.21.15.45:0/2122771864 wait complete. 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 10 monclient: build_initial_monmap 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 10 monclient: monmap: 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:35.003 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 librados: starting msgr at 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 librados: starting objecter 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 Processor -- start 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 -- start start 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 librados: setting wanted keys 2024-03-29T11:33:35.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 1 librados: calling monclient init 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.998+0000 7f4c291fe640 10 monclient: init 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.999+0000 7f4c291fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.999+0000 7f4c291fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:34.999+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2406c250 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4c240ae650 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2413f8c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c2259c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4c240ae650 0x7f4c241d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59778/0 (socket says 172.21.15.45:59778) 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c2259c640 1 -- 172.21.15.45:0/1526086337 learned_addr learned my addr 172.21.15.45:0/1526086337 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f4c240ae650 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:35.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240729a0 con 0x7f4c2413f8c0 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240724b0 con 0x7f4c2406c250 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c24071620 con 0x7f4c240ae650 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.000+0000 7f4c291fe640 10 monclient(hunting): authenticate will time out at 871.239948s 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788567981 0 0) 0x7f4c24071620 con 0x7f4c240ae650 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:33:35.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10004eb0 con 0x7f4c240ae650 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 client.?.objecter ms_handle_connect 0x7f4c2406c250 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c22d9d640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6789/0 conn(0x7f4c2406c250 legacy=0x7f4c24145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c22d9d640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6789/0 conn(0x7f4c2406c250 legacy=0x7f4c24145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c22d9d640 1 --1- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6789/0 conn(0x7f4c2406c250 0x7f4c24145ac0 :-1 s=OPENED pgs=189832 cs=1 l=1).handle_message read tag failed 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c22d9d640 1 --1- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6789/0 conn(0x7f4c2406c250 0x7f4c24145ac0 :-1 s=OPENED pgs=189832 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.190:6789/0 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142503621 0 0) 0x7f4c10004eb0 con 0x7f4c240ae650 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c24071620 con 0x7f4c240ae650 2024-03-29T11:33:35.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4c14001960 con 0x7f4c240ae650 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): dump: 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:35.008 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419164300 0 0) 0x7f4c24071620 con 0x7f4c240ae650 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6789/0 conn(0x7f4c2406c250 legacy=0x7f4c24145ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.003+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.45:6789/0 conn(0x7f4c2413f8c0 legacy=0x7f4c241d7de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 1 monclient: found mon.c 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24141810 con 0x7f4c240ae650 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:35.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 10 monclient: _renew_subs 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c24141ae0 con 0x7f4c240ae650 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c14001d40 con 0x7f4c240ae650 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4c14003a40 con 0x7f4c240ae650 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c0b7fe640 10 monclient: dump: 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:35.010 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 10 monclient: _renew_subs 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c24072070 con 0x7f4c240ae650 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c291fe640 1 librados: init done 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.004+0000 7f4c2159a640 4 set_mon_vals no callback set 2024-03-29T11:33:35.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.006+0000 7f4c097fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:35.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.006+0000 7f4c097fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:35.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.006+0000 7f4c097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:35.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.006+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c240726f0 con 0x7f4c240ae650 2024-03-29T11:33:35.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.006+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f4c14003ba0 con 0x7f4c240ae650 2024-03-29T11:33:35.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.007+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f4c14084b50 con 0x7f4c240ae650 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.007+0000 7f4c0b7fe640 10 client.4286.objecter ms_dispatch 0x7f4c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.007+0000 7f4c0b7fe640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.007+0000 7f4c0b7fe640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.007+0000 7f4c0b7fe640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.008+0000 7f4c22d9d640 10 monclient: get_auth_request con 0x7f4c1006dc40 auth_method 0 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.008+0000 7f4c0b7fe640 10 client.4286.objecter ms_handle_connect 0x7f4c1006dc40 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.014+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c240726f0 con 0x7f4c240ae650 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.014+0000 7f4c0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:35.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.014+0000 7f4c0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:35.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.327+0000 7f4c097fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:35.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.327+0000 7f4c097fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:35.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.327+0000 7f4c097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:35.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.327+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c240726f0 con 0x7f4c240ae650 2024-03-29T11:33:35.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.328+0000 7f4c0b7fe640 1 -- 172.21.15.45:0/1526086337 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f4c240726f0 con 0x7f4c240ae650 2024-03-29T11:33:35.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.328+0000 7f4c0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:35.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.328+0000 7f4c0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:35.325 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:35.325 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:35.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 10 librados: watch_flush enter 2024-03-29T11:33:35.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 10 librados: watch_flush exit 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 10 client.4286.objecter successfully canceled tick 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.190:6816/1411598462 conn(0x7f4c1006dc40 legacy=0x7f4c10070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 10 monclient: shutdown 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.331+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 >> v1:172.21.15.45:6790/0 conn(0x7f4c240ae650 legacy=0x7f4c241d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.332+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 shutdown_connections 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.332+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 >> 172.21.15.45:0/1526086337 conn(0x7f4c24067610 msgr2=0x7f4c2413ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.332+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 shutdown_connections 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.332+0000 7f4c097fa640 1 -- 172.21.15.45:0/1526086337 wait complete. 2024-03-29T11:33:35.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:35.332+0000 7f4c097fa640 1 librados: shutdown 2024-03-29T11:33:35.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:42.152 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:42.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.303+0000 7f8f105d6640 10 monclient: get_monmap_and_config 2024-03-29T11:33:42.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.303+0000 7f8f105d6640 10 monclient: build_initial_monmap 2024-03-29T11:33:42.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.303+0000 7f8f105d6640 10 monclient: monmap: 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:42.304789+0000 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:42.304789+0000 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:42.301 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:42.302 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:42.302 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.304+0000 7f8f105d6640 1 Processor -- start 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.304+0000 7f8f105d6640 1 -- start start 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.304+0000 7f8f105d6640 10 monclient: init 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.305+0000 7f8f105d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.305+0000 7f8f105d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.305+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f081457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.305+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f080a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080aaef0 con 0x7f8f0813f650 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08145bd0 con 0x7f8f081457f0 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08145eb0 con 0x7f8f080a9d70 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:42.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.306+0000 7f8f105d6640 10 monclient(hunting): authenticate will time out at 878.545140s 2024-03-29T11:33:42.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.307+0000 7f8f0eb73640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8f080a9d70 0x7f8f0814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38418/0 (socket says 172.21.15.45:38418) 2024-03-29T11:33:42.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.307+0000 7f8f0eb73640 1 -- 172.21.15.45:0/2317395188 learned_addr learned my addr 172.21.15.45:0/2317395188 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493146790 0 0) 0x7f8f08145eb0 con 0x7f8f080a9d70 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 10 monclient(hunting): _negotiate 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef4003520 con 0x7f8f080a9d70 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.308+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256747765 0 0) 0x7f8ef4003520 con 0x7f8f080a9d70 2024-03-29T11:33:42.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.309+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 >> v1:172.21.15.190:6789/0 conn(0x7f8f081457f0 legacy=0x7f8f0813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 >> v1:172.21.15.45:6789/0 conn(0x7f8f0813f650 legacy=0x7f8f080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: _finish_hunting 0 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 1 monclient: found mon.noname-b 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08146340 con 0x7f8f080a9d70 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: _finish_auth 0 2024-03-29T11:33:42.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: _check_auth_tickets 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8f00001a10 con 0x7f8f080a9d70 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f0d370640 10 monclient: dump: 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:42.308 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f105d6640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:33:42.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.310+0000 7f8f105d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:42.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.311+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f00001be0 con 0x7f8f080a9d70 2024-03-29T11:33:42.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f0d370640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:42.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f0d370640 1 -- 172.21.15.45:0/2317395188 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8f00002030 con 0x7f8f080a9d70 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f0d370640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f0d370640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f0d370640 10 monclient: dump: 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f105d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f105d6640 4 set_mon_vals no callback set 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f105d6640 10 monclient: shutdown 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f105d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.312+0000 7f8f105d6640 1 -- 172.21.15.45:0/2317395188 >> v1:172.21.15.45:6790/0 conn(0x7f8f080a9d70 legacy=0x7f8f0814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.313+0000 7f8f105d6640 1 -- 172.21.15.45:0/2317395188 shutdown_connections 2024-03-29T11:33:42.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.313+0000 7f8f105d6640 1 -- 172.21.15.45:0/2317395188 >> 172.21.15.45:0/2317395188 conn(0x7f8f08066a00 msgr2=0x7f8f08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.315+0000 7f8f105d6640 1 -- 172.21.15.45:0/2317395188 shutdown_connections 2024-03-29T11:33:42.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.315+0000 7f8f105d6640 1 -- 172.21.15.45:0/2317395188 wait complete. 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.316+0000 7f8f105d6640 10 monclient: build_initial_monmap 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.316+0000 7f8f105d6640 10 monclient: monmap: 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.314 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 librados: starting msgr at 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 librados: starting objecter 2024-03-29T11:33:42.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 Processor -- start 2024-03-29T11:33:42.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 -- start start 2024-03-29T11:33:42.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 librados: setting wanted keys 2024-03-29T11:33:42.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 1 librados: calling monclient init 2024-03-29T11:33:42.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.317+0000 7f8f105d6640 10 monclient: init 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f080a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f0813f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f081457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0813b350 con 0x7f8f080a9d70 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0806ee10 con 0x7f8f0813f650 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0806f0f0 con 0x7f8f081457f0 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.318+0000 7f8f105d6640 10 monclient(hunting): authenticate will time out at 878.557830s 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8f0eb73640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8f081457f0 0x7f8f081eeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39098/0 (socket says 172.21.15.45:39098) 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8f0eb73640 1 -- 172.21.15.45:0/2514164240 learned_addr learned my addr 172.21.15.45:0/2514164240 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 client.?.objecter ms_handle_connect 0x7f8f081457f0 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:42.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053806601 0 0) 0x7f8f0806f0f0 con 0x7f8f081457f0 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.319+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eec004eb0 con 0x7f8f081457f0 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033629526 0 0) 0x7f8eec004eb0 con 0x7f8f081457f0 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f0806f0f0 con 0x7f8f081457f0 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8f00001ef0 con 0x7f8f081457f0 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 10 monclient(hunting): dump: 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.320 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.320+0000 7f8efeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.322+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186546901 0 0) 0x7f8f0806f0f0 con 0x7f8f081457f0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.322+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 >> v1:172.21.15.190:6789/0 conn(0x7f8f0813f650 legacy=0x7f8f081395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.322+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 >> v1:172.21.15.45:6789/0 conn(0x7f8f080a9d70 legacy=0x7f8f08138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.322+0000 7f8efeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 1 monclient: found mon.c 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0806f3d0 con 0x7f8f081457f0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:42.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 10 monclient: _renew_subs 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0813bb50 con 0x7f8f081457f0 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 10 monclient: _renew_subs 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0806e2e0 con 0x7f8f081457f0 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.323+0000 7f8f105d6640 1 librados: init done 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.325+0000 7f8f105d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.325+0000 7f8f105d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.325+0000 7f8f105d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.325+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f08075f10 con 0x7f8f081457f0 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f00004f70 con 0x7f8f081457f0 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8f00005280 con 0x7f8f081457f0 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efeffd640 10 monclient: dump: 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:42.325 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:42.326 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:42.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.326+0000 7f8efffff640 4 set_mon_vals no callback set 2024-03-29T11:33:42.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.327+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f8f0001b250 con 0x7f8f081457f0 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.328+0000 7f8f0e372640 10 monclient: get_auth_request con 0x7f8eec06dc70 auth_method 0 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.330+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f8f00084f70 con 0x7f8f081457f0 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.330+0000 7f8efeffd640 10 client.4292.objecter ms_dispatch 0x7f8f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.330+0000 7f8efeffd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.330+0000 7f8efeffd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.330+0000 7f8efeffd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:33:42.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.331+0000 7f8efeffd640 10 client.4292.objecter ms_handle_connect 0x7f8eec06dc70 2024-03-29T11:33:42.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.333+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f00050850 con 0x7f8f081457f0 2024-03-29T11:33:42.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.333+0000 7f8efeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:42.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.333+0000 7f8efeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:42.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.649+0000 7f8f105d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:42.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.650+0000 7f8f105d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:42.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.650+0000 7f8f105d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:42.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.650+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f080b3680 con 0x7f8f081457f0 2024-03-29T11:33:42.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.651+0000 7f8efeffd640 1 -- 172.21.15.45:0/2514164240 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f8f00052780 con 0x7f8f081457f0 2024-03-29T11:33:42.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.651+0000 7f8efeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:42.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.651+0000 7f8efeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:42.648 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:42.648 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:42.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.654+0000 7f8f105d6640 10 librados: watch_flush enter 2024-03-29T11:33:42.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.654+0000 7f8f105d6640 10 librados: watch_flush exit 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.654+0000 7f8f105d6640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.654+0000 7f8f105d6640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 >> v1:172.21.15.190:6816/1411598462 conn(0x7f8eec06dc70 legacy=0x7f8eec070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 10 monclient: shutdown 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 >> v1:172.21.15.45:6790/0 conn(0x7f8f081457f0 legacy=0x7f8f081eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 shutdown_connections 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 >> 172.21.15.45:0/2514164240 conn(0x7f8f08066a00 msgr2=0x7f8f08066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 shutdown_connections 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 -- 172.21.15.45:0/2514164240 wait complete. 2024-03-29T11:33:42.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:42.655+0000 7f8f105d6640 1 librados: shutdown 2024-03-29T11:33:42.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:49.474 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.624+0000 7f63704c8640 10 monclient: get_monmap_and_config 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.624+0000 7f63704c8640 10 monclient: build_initial_monmap 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.624+0000 7f63704c8640 10 monclient: monmap: 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:49.625991+0000 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:49.625991+0000 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:49.622 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:49.623 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.625+0000 7f63704c8640 1 Processor -- start 2024-03-29T11:33:49.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.626+0000 7f63704c8640 1 -- start start 2024-03-29T11:33:49.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.626+0000 7f63704c8640 10 monclient: init 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-a con 0x7f63680b0860 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-b con 0x7f636813ef30 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-c con 0x7f63680af8e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680b0c40 con 0x7f63680b0860 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680aed10 con 0x7f63680af8e0 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680afd50 con 0x7f636813ef30 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:49.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.627+0000 7f63704c8640 10 monclient(hunting): authenticate will time out at 885.866333s 2024-03-29T11:33:49.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.628+0000 7f636ea65640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f63680af8e0 0x7f6368149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45648/0 (socket says 172.21.15.45:45648) 2024-03-29T11:33:49.625 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.628+0000 7f636ea65640 1 -- 172.21.15.45:0/2373426581 learned_addr learned my addr 172.21.15.45:0/2373426581 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:49.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.629+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751605030 0 0) 0x7f63680aed10 con 0x7f63680af8e0 2024-03-29T11:33:49.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.629+0000 7f636d262640 10 monclient(hunting): _negotiate 2024-03-29T11:33:49.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.629+0000 7f636d262640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:49.626 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.630+0000 7f636d262640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:49.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.630+0000 7f636d262640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:33:49.627 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.630+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6354003520 con 0x7f63680af8e0 2024-03-29T11:33:49.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.631+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1388039617 0 0) 0x7f6354003520 con 0x7f63680af8e0 2024-03-29T11:33:49.628 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.631+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 >> v1:172.21.15.45:6790/0 conn(0x7f636813ef30 legacy=0x7f63680af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 >> v1:172.21.15.45:6789/0 conn(0x7f63680b0860 legacy=0x7f636813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: _finish_hunting 0 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 monclient: found mon.noname-c 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.629 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63680ab1b0 con 0x7f63680af8e0 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: _finish_auth 0 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: _check_auth_tickets 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6360001a10 con 0x7f63680af8e0 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: dump: 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.630 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6360001ca0 con 0x7f63680af8e0 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 1 -- 172.21.15.45:0/2373426581 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6360002030 con 0x7f63680af8e0 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.632+0000 7f636d262640 10 monclient: dump: 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:49.631 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.632 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:49.632 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:49.632 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:49.632 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.633+0000 7f63704c8640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.633+0000 7f63704c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.633+0000 7f63704c8640 4 set_mon_vals no callback set 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 10 monclient: shutdown 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 1 -- 172.21.15.45:0/2373426581 >> v1:172.21.15.190:6789/0 conn(0x7f63680af8e0 legacy=0x7f6368149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 1 -- 172.21.15.45:0/2373426581 shutdown_connections 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 1 -- 172.21.15.45:0/2373426581 >> 172.21.15.45:0/2373426581 conn(0x7f636806b660 msgr2=0x7f636806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 1 -- 172.21.15.45:0/2373426581 shutdown_connections 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.634+0000 7f63704c8640 1 -- 172.21.15.45:0/2373426581 wait complete. 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.635+0000 7f63704c8640 10 monclient: build_initial_monmap 2024-03-29T11:33:49.635 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.635+0000 7f63704c8640 10 monclient: monmap: 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.635+0000 7f63704c8640 1 librados: starting msgr at 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.635+0000 7f63704c8640 1 librados: starting objecter 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 1 Processor -- start 2024-03-29T11:33:49.636 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 1 -- start start 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 1 librados: setting wanted keys 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 1 librados: calling monclient init 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 10 monclient: init 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-a con 0x7f63680af8e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.636+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-c con 0x7f63680b0860 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 10 monclient(hunting): picked mon.noname-b con 0x7f636813ef30 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680b3c00 con 0x7f63680af8e0 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680b38c0 con 0x7f636813ef30 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63680b2930 con 0x7f63680b0860 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:49.637 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.637+0000 7f63704c8640 10 monclient(hunting): authenticate will time out at 885.876757s 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.639+0000 7f636ea65640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f636813ef30 0x7f63681d3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:47198/0 (socket says 172.21.15.45:47198) 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.639+0000 7f636ea65640 1 -- 172.21.15.45:0/1931748454 learned_addr learned my addr 172.21.15.45:0/1931748454 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter ms_handle_connect 0x7f63680af8e0 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter ms_handle_connect 0x7f636813ef30 2024-03-29T11:33:49.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752992889 0 0) 0x7f63680b38c0 con 0x7f636813ef30 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f636813ef30 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter ms_handle_connect 0x7f63680b0860 2024-03-29T11:33:49.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885613523 0 0) 0x7f634c004eb0 con 0x7f636813ef30 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63680b38c0 con 0x7f636813ef30 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f63600018d0 con 0x7f636813ef30 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.641+0000 7f635effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): dump: 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.641 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817325535 0 0) 0x7f63680b2930 con 0x7f63680b0860 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:49.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f63680b0860 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336069622 0 0) 0x7f63680b3c00 con 0x7f63680af8e0 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63680b2930 con 0x7f63680af8e0 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 654966112 0 0) 0x7f63680b38c0 con 0x7f636813ef30 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 >> v1:172.21.15.45:6790/0 conn(0x7f63680b0860 legacy=0x7f63681d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 >> v1:172.21.15.45:6789/0 conn(0x7f63680af8e0 legacy=0x7f63680b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 monclient: found mon.b 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:49.643 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63681376a0 con 0x7f636813ef30 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient: _finish_auth 0 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f635effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 10 monclient: _renew_subs 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63681378f0 con 0x7f636813ef30 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.642+0000 7f63704c8640 10 monclient: _renew_subs 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f63704c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63681dec50 con 0x7f636813ef30 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f63704c8640 1 librados: init done 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6360004f70 con 0x7f636813ef30 2024-03-29T11:33:49.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6360005280 con 0x7f636813ef30 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f636ca61640 4 set_mon_vals no callback set 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.643+0000 7f635effd640 10 monclient: dump: 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:49.645 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:49.646 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.645+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f636001b250 con 0x7f636813ef30 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.646+0000 7f636e264640 10 monclient: get_auth_request con 0x7f634c06deb0 auth_method 0 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.646+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f6360084ed0 con 0x7f636813ef30 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.646+0000 7f635effd640 10 client.4270.objecter ms_dispatch 0x7f6368000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.646+0000 7f635effd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:49.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.646+0000 7f635effd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f635effd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f635effd640 10 client.4270.objecter ms_handle_connect 0x7f634c06deb0 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f63704c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f63704c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f63704c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.647+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6350005080 con 0x7f636813ef30 2024-03-29T11:33:49.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.655+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63600507b0 con 0x7f636813ef30 2024-03-29T11:33:49.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.656+0000 7f635effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:49.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.656+0000 7f635effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:49.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.990+0000 7f63704c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:49.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.990+0000 7f63704c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:49.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.990+0000 7f63704c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:33:49.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.991+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6350005330 con 0x7f636813ef30 2024-03-29T11:33:49.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.994+0000 7f635effd640 1 -- 172.21.15.45:0/1931748454 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f63600526e0 con 0x7f636813ef30 2024-03-29T11:33:49.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.994+0000 7f635effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:49.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.994+0000 7f635effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:49.991 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:49.991 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:49.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.997+0000 7f63704c8640 10 librados: watch_flush enter 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 10 librados: watch_flush exit 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 >> v1:172.21.15.190:6816/1411598462 conn(0x7f634c06deb0 legacy=0x7f634c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 10 monclient: shutdown 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 >> v1:172.21.15.190:6789/0 conn(0x7f636813ef30 legacy=0x7f63681d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:49.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 shutdown_connections 2024-03-29T11:33:49.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.998+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 >> 172.21.15.45:0/1931748454 conn(0x7f636806b660 msgr2=0x7f636813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:49.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.999+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 shutdown_connections 2024-03-29T11:33:49.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.999+0000 7f63704c8640 1 -- 172.21.15.45:0/1931748454 wait complete. 2024-03-29T11:33:49.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:49.999+0000 7f63704c8640 1 librados: shutdown 2024-03-29T11:33:50.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:56.821 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.977+0000 7f85866ac640 10 monclient: get_monmap_and_config 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.977+0000 7f85866ac640 10 monclient: build_initial_monmap 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.977+0000 7f85866ac640 10 monclient: monmap: 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:33:56.977286+0000 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:33:56.977286+0000 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:56.973 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:33:56.974 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:33:56.974 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:56.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.979+0000 7f85866ac640 1 Processor -- start 2024-03-29T11:33:56.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.979+0000 7f85866ac640 1 -- start start 2024-03-29T11:33:56.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.979+0000 7f85866ac640 10 monclient: init 2024-03-29T11:33:56.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:56.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-b con 0x7f8578006e30 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-c con 0x7f8578035e00 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-a con 0x7f85780c9670 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578035970 con 0x7f85780c9670 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578036280 con 0x7f8578035e00 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578035640 con 0x7f8578006e30 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.980+0000 7f85866ac640 10 monclient(hunting): authenticate will time out at 893.218961s 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8585eab640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f85780c9670 0x7f85780d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43371/0 (socket says 172.21.15.45:43371) 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8585eab640 1 -- 172.21.15.45:0/2452384735 learned_addr learned my addr 172.21.15.45:0/2452384735 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867277077 0 0) 0x7f8578035970 con 0x7f85780c9670 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:56.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.984+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c003520 con 0x7f85780c9670 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104943884 0 0) 0x7f856c003520 con 0x7f85780c9670 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 >> v1:172.21.15.45:6790/0 conn(0x7f8578006e30 legacy=0x7f85780cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 >> v1:172.21.15.190:6789/0 conn(0x7f8578035e00 legacy=0x7f85780c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 monclient: found mon.noname-a 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578034e20 con 0x7f85780c9670 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:56.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f858004f730 con 0x7f85780c9670 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f8577fff640 10 monclient: dump: 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:56.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f85866ac640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:33:56.985 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.985+0000 7f85866ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.988+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85800008d0 con 0x7f85780c9670 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f8577fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f8577fff640 1 -- 172.21.15.45:0/2452384735 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f85800758b0 con 0x7f85780c9670 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f8577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f8577fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f8577fff640 10 monclient: dump: 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:56.986 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f85866ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.989+0000 7f85866ac640 4 set_mon_vals no callback set 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.990+0000 7f85866ac640 10 monclient: shutdown 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.990+0000 7f85866ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.990+0000 7f85866ac640 1 -- 172.21.15.45:0/2452384735 >> v1:172.21.15.45:6789/0 conn(0x7f85780c9670 legacy=0x7f85780d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.990+0000 7f85866ac640 1 -- 172.21.15.45:0/2452384735 shutdown_connections 2024-03-29T11:33:56.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.990+0000 7f85866ac640 1 -- 172.21.15.45:0/2452384735 >> 172.21.15.45:0/2452384735 conn(0x7f8578005c00 msgr2=0x7f8578006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.991+0000 7f85866ac640 1 -- 172.21.15.45:0/2452384735 shutdown_connections 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.991+0000 7f85866ac640 1 -- 172.21.15.45:0/2452384735 wait complete. 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.992+0000 7f85866ac640 10 monclient: build_initial_monmap 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.992+0000 7f85866ac640 10 monclient: monmap: 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:33:56.988 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.992+0000 7f85866ac640 1 librados: starting msgr at 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.992+0000 7f85866ac640 1 librados: starting objecter 2024-03-29T11:33:56.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.992+0000 7f85866ac640 1 Processor -- start 2024-03-29T11:33:56.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.994+0000 7f85866ac640 1 -- start start 2024-03-29T11:33:56.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.994+0000 7f85866ac640 1 librados: setting wanted keys 2024-03-29T11:33:56.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.994+0000 7f85866ac640 1 librados: calling monclient init 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.994+0000 7f85866ac640 10 monclient: init 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-c con 0x7f8578006e30 addr v1:172.21.15.45:6790/0 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-a con 0x7f8578035e00 addr v1:172.21.15.45:6789/0 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient(hunting): picked mon.noname-b con 0x7f85780c9670 addr v1:172.21.15.190:6789/0 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578160d80 con 0x7f8578035e00 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85781619b0 con 0x7f85780c9670 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578161b40 con 0x7f8578006e30 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:56.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.995+0000 7f85866ac640 10 monclient(hunting): authenticate will time out at 893.233275s 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.998+0000 7f85856aa640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8578006e30 0x7f85781612a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55153/0 (socket says 172.21.15.45:55153) 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:56.998+0000 7f85856aa640 1 -- 172.21.15.45:0/3369488201 learned_addr learned my addr 172.21.15.45:0/3369488201 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 client.?.objecter ms_handle_connect 0x7f8578006e30 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 client.?.objecter ms_handle_connect 0x7f8578035e00 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531870042 0 0) 0x7f8578161b40 con 0x7f8578006e30 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:33:56.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8564004eb0 con 0x7f8578006e30 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312571363 0 0) 0x7f8578160d80 con 0x7f8578035e00 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.001+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8578161b40 con 0x7f8578035e00 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583425291 0 0) 0x7f8564004eb0 con 0x7f8578006e30 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8578160d80 con 0x7f8578006e30 2024-03-29T11:33:56.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f857c001960 con 0x7f8578006e30 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 10 monclient(hunting): dump: 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:57.002 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.002+0000 7f8575ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108378102 0 0) 0x7f8578161b40 con 0x7f8578035e00 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8564004eb0 con 0x7f8578035e00 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2087202870 0 0) 0x7f8578160d80 con 0x7f8578006e30 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 >> v1:172.21.15.190:6789/0 conn(0x7f85780c9670 legacy=0x7f857802f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 >> v1:172.21.15.45:6789/0 conn(0x7f8578035e00 legacy=0x7f85780ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 monclient: found mon.c 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780cec70 con 0x7f8578006e30 2024-03-29T11:33:57.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.004+0000 7f8575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 10 monclient: _renew_subs 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8578031e30 con 0x7f8578006e30 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 10 monclient: _renew_subs 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8578030100 con 0x7f8578006e30 2024-03-29T11:33:57.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.005+0000 7f85866ac640 1 librados: init done 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c0035f0 con 0x7f8578006e30 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f857c003940 con 0x7f8578006e30 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.006+0000 7f8575ffb640 10 monclient: dump: 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:33:57.006 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.007+0000 7f85866ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.007+0000 7f85866ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.007+0000 7f85866ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.007+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8578007b90 con 0x7f8578006e30 2024-03-29T11:33:57.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.008+0000 7f8576ffd640 4 set_mon_vals no callback set 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.011+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f857c003aa0 con 0x7f8578006e30 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f857c084990 con 0x7f8578006e30 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 10 client.4298.objecter ms_dispatch 0x7f8580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:57.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:33:57.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8584ea9640 10 monclient: get_auth_request con 0x7f856406df20 auth_method 0 2024-03-29T11:33:57.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.012+0000 7f8575ffb640 10 client.4298.objecter ms_handle_connect 0x7f856406df20 2024-03-29T11:33:57.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.017+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f857c050270 con 0x7f8578006e30 2024-03-29T11:33:57.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.017+0000 7f8575ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:57.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.017+0000 7f8575ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:57.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.345+0000 7f85866ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:57.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.346+0000 7f85866ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:57.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.346+0000 7f85866ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:33:57.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.346+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8578000b90 con 0x7f8578006e30 2024-03-29T11:33:57.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.348+0000 7f8575ffb640 1 -- 172.21.15.45:0/3369488201 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f8578000b90 con 0x7f8578006e30 2024-03-29T11:33:57.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.349+0000 7f8575ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:57.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.349+0000 7f8575ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:57.345 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:33:57.345 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:57.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 10 librados: watch_flush enter 2024-03-29T11:33:57.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 10 librados: watch_flush exit 2024-03-29T11:33:57.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:33:57.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:33:57.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 >> v1:172.21.15.190:6816/1411598462 conn(0x7f856406df20 legacy=0x7f85640703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 10 monclient: shutdown 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 >> v1:172.21.15.45:6790/0 conn(0x7f8578006e30 legacy=0x7f85781612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.353+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 shutdown_connections 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.354+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 >> 172.21.15.45:0/3369488201 conn(0x7f8578005c00 msgr2=0x7f85780c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.354+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 shutdown_connections 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.354+0000 7f85866ac640 1 -- 172.21.15.45:0/3369488201 wait complete. 2024-03-29T11:33:57.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:33:57.354+0000 7f85866ac640 1 librados: shutdown 2024-03-29T11:33:57.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:04.174 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.329+0000 7f55e913b640 10 monclient: get_monmap_and_config 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.329+0000 7f55e913b640 10 monclient: build_initial_monmap 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.329+0000 7f55e913b640 10 monclient: monmap: 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:04.330262+0000 2024-03-29T11:34:04.326 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:04.330262+0000 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.330+0000 7f55e913b640 1 Processor -- start 2024-03-29T11:34:04.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.333+0000 7f55e913b640 1 -- start start 2024-03-29T11:34:04.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.333+0000 7f55e913b640 10 monclient: init 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-a con 0x7f55e40b4920 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-b con 0x7f55e40b4200 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-c con 0x7f55e413f9a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40b0930 con 0x7f55e40b4920 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40af460 con 0x7f55e413f9a0 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40aeda0 con 0x7f55e40b4200 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:04.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.334+0000 7f55e913b640 10 monclient(hunting): authenticate will time out at 900.574020s 2024-03-29T11:34:04.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.338+0000 7f55e3fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f55e40b4920 0x7f55e413f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56457/0 (socket says 172.21.15.45:56457) 2024-03-29T11:34:04.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.338+0000 7f55e3fff640 1 -- 172.21.15.45:0/1422035153 learned_addr learned my addr 172.21.15.45:0/1422035153 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:04.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.339+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837603937 0 0) 0x7f55e40aeda0 con 0x7f55e40b4200 2024-03-29T11:34:04.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:04.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:04.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:04.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55cc003520 con 0x7f55e40b4200 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746352467 0 0) 0x7f55cc003520 con 0x7f55e40b4200 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 >> v1:172.21.15.190:6789/0 conn(0x7f55e413f9a0 legacy=0x7f55e414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 >> v1:172.21.15.45:6789/0 conn(0x7f55e40b4920 legacy=0x7f55e413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.340+0000 7f55e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e40af980 con 0x7f55e40b4200 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f55d8001690 con 0x7f55e40b4200 2024-03-29T11:34:04.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: dump: 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55d8001920 con 0x7f55e40b4200 2024-03-29T11:34:04.341 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.341+0000 7f55e2ffd640 1 -- 172.21.15.45:0/1422035153 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f55d8001690 con 0x7f55e40b4200 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e2ffd640 10 monclient: dump: 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:04.342 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e913b640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:34:04.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e913b640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:04.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.342+0000 7f55e913b640 4 set_mon_vals no callback set 2024-03-29T11:34:04.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 10 monclient: shutdown 2024-03-29T11:34:04.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 1 -- 172.21.15.45:0/1422035153 >> v1:172.21.15.45:6790/0 conn(0x7f55e40b4200 legacy=0x7f55e40b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 1 -- 172.21.15.45:0/1422035153 shutdown_connections 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 1 -- 172.21.15.45:0/1422035153 >> 172.21.15.45:0/1422035153 conn(0x7f55e406d230 msgr2=0x7f55e4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.343+0000 7f55e913b640 1 -- 172.21.15.45:0/1422035153 shutdown_connections 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.344+0000 7f55e913b640 1 -- 172.21.15.45:0/1422035153 wait complete. 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.344+0000 7f55e913b640 10 monclient: build_initial_monmap 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.344+0000 7f55e913b640 10 monclient: monmap: 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:04.345 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.345+0000 7f55e913b640 1 librados: starting msgr at 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.345+0000 7f55e913b640 1 librados: starting objecter 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.345+0000 7f55e913b640 1 Processor -- start 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.347+0000 7f55e913b640 1 -- start start 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.347+0000 7f55e913b640 1 librados: setting wanted keys 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.347+0000 7f55e913b640 1 librados: calling monclient init 2024-03-29T11:34:04.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.347+0000 7f55e913b640 10 monclient: init 2024-03-29T11:34:04.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.348+0000 7f55e913b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:04.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.348+0000 7f55e913b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:04.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.348+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-a con 0x7f55e40b4200 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-c con 0x7f55e40b4920 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 10 monclient(hunting): picked mon.noname-b con 0x7f55e413f9a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e41d6f50 con 0x7f55e40b4200 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40ad7b0 con 0x7f55e413f9a0 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e41d7af0 con 0x7f55e40b4920 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:04.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.349+0000 7f55e913b640 10 monclient(hunting): authenticate will time out at 900.588649s 2024-03-29T11:34:04.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e893a640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f55e413f9a0 0x7f55e4139fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:55401/0 (socket says 172.21.15.45:55401) 2024-03-29T11:34:04.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e893a640 1 -- 172.21.15.45:0/857222120 learned_addr learned my addr 172.21.15.45:0/857222120 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:04.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e0ff9640 10 client.?.objecter ms_handle_connect 0x7f55e40b4200 2024-03-29T11:34:04.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:04.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e0ff9640 10 client.?.objecter ms_handle_connect 0x7f55e413f9a0 2024-03-29T11:34:04.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.353+0000 7f55e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:04.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338667615 0 0) 0x7f55e41d6f50 con 0x7f55e40b4200 2024-03-29T11:34:04.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:04.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:04.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:04.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d0004eb0 con 0x7f55e40b4200 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311986525 0 0) 0x7f55e40ad7b0 con 0x7f55e413f9a0 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:34:04.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e41d6f50 con 0x7f55e413f9a0 2024-03-29T11:34:04.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 client.?.objecter ms_handle_connect 0x7f55e40b4920 2024-03-29T11:34:04.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.357+0000 7f55e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950457114 0 0) 0x7f55d0004eb0 con 0x7f55e40b4200 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55e40ad7b0 con 0x7f55e40b4200 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f55d4001930 con 0x7f55e40b4200 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.358 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1656902867 0 0) 0x7f55e41d6f50 con 0x7f55e413f9a0 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55d0004eb0 con 0x7f55e413f9a0 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f55dc001a90 con 0x7f55e413f9a0 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.359 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255032816 0 0) 0x7f55e41d7af0 con 0x7f55e40b4920 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.358+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e41d6f50 con 0x7f55e40b4920 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 176542106 0 0) 0x7f55e40ad7b0 con 0x7f55e40b4200 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 >> v1:172.21.15.45:6790/0 conn(0x7f55e40b4920 legacy=0x7f55e41d73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 >> v1:172.21.15.190:6789/0 conn(0x7f55e413f9a0 legacy=0x7f55e4139fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 1 monclient: found mon.a 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e41d7c80 con 0x7f55e40b4200 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 10 monclient: _renew_subs 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e413a6c0 con 0x7f55e40b4200 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:04.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 10 monclient: _renew_subs 2024-03-29T11:34:04.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:04.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e40aafc0 con 0x7f55e40b4200 2024-03-29T11:34:04.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.359+0000 7f55e913b640 1 librados: init done 2024-03-29T11:34:04.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.361+0000 7f55ca7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:04.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.361+0000 7f55ca7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:04.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.362+0000 7f55ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.362+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e41417f0 con 0x7f55e40b4200 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55d4001d60 con 0x7f55e40b4200 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f55d4004980 con 0x7f55e40b4200 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.363+0000 7f55e0ff9640 10 monclient: dump: 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:04.364 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.364+0000 7f55e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.365+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f55d4004b60 con 0x7f55e40b4200 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.365+0000 7f55e37fe640 10 monclient: get_auth_request con 0x7f55d006e130 auth_method 0 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f55d4054670 con 0x7f55e40b4200 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 10 client.4305.objecter ms_dispatch 0x7f55e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-29T11:34:04.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.366+0000 7f55e0ff9640 10 client.4305.objecter ms_handle_connect 0x7f55d006e130 2024-03-29T11:34:04.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.376+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55d4050910 con 0x7f55e40b4200 2024-03-29T11:34:04.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.376+0000 7f55e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:04.373 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.376+0000 7f55e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:04.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.704+0000 7f55ca7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:04.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.705+0000 7f55ca7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:04.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.705+0000 7f55ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:04.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.705+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55e404f730 con 0x7f55e40b4200 2024-03-29T11:34:04.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.706+0000 7f55e0ff9640 1 -- 172.21.15.45:0/857222120 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f55d4050910 con 0x7f55e40b4200 2024-03-29T11:34:04.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.706+0000 7f55e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:04.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.706+0000 7f55e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:04.706 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:04.706 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 10 librados: watch_flush enter 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 10 librados: watch_flush exit 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 10 client.4305.objecter successfully canceled tick 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 >> v1:172.21.15.190:6816/1411598462 conn(0x7f55d006e130 legacy=0x7f55d00705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 10 monclient: shutdown 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:04.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.712+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 >> v1:172.21.15.45:6789/0 conn(0x7f55e40b4200 legacy=0x7f55e40adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:04.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.713+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 shutdown_connections 2024-03-29T11:34:04.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.713+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 >> 172.21.15.45:0/857222120 conn(0x7f55e406d230 msgr2=0x7f55e413bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:04.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.714+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 shutdown_connections 2024-03-29T11:34:04.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.715+0000 7f55ca7fc640 1 -- 172.21.15.45:0/857222120 wait complete. 2024-03-29T11:34:04.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:04.715+0000 7f55ca7fc640 1 librados: shutdown 2024-03-29T11:34:04.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:11.535 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:11.695 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.693+0000 7fe44eae2640 10 monclient: get_monmap_and_config 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.693+0000 7fe44eae2640 10 monclient: build_initial_monmap 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.693+0000 7fe44eae2640 10 monclient: monmap: 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:11.694506+0000 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:11.694506+0000 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:11.696 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.694+0000 7fe44eae2640 1 Processor -- start 2024-03-29T11:34:11.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.694+0000 7fe44eae2640 1 -- start start 2024-03-29T11:34:11.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.694+0000 7fe44eae2640 10 monclient: init 2024-03-29T11:34:11.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:11.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:11.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4481457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:11.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-a con 0x7fe44813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4480a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480aaef0 con 0x7fe44813f650 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe448145bd0 con 0x7fe4480a9d70 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe448145eb0 con 0x7fe4481457f0 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:11.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.695+0000 7fe44eae2640 10 monclient(hunting): authenticate will time out at 907.935059s 2024-03-29T11:34:11.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.699+0000 7fe44d07f640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe4480a9d70 0x7fe44814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:55467/0 (socket says 172.21.15.45:55467) 2024-03-29T11:34:11.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.699+0000 7fe44d07f640 1 -- 172.21.15.45:0/1402053114 learned_addr learned my addr 172.21.15.45:0/1402053114 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:11.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930354269 0 0) 0x7fe448145bd0 con 0x7fe4480a9d70 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430003520 con 0x7fe4480a9d70 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240688561 0 0) 0x7fe448145eb0 con 0x7fe4481457f0 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:11.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.701+0000 7fe4477fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe448145bd0 con 0x7fe4481457f0 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154185647 0 0) 0x7fe430003520 con 0x7fe4480a9d70 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 >> v1:172.21.15.45:6790/0 conn(0x7fe4481457f0 legacy=0x7fe44813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 >> v1:172.21.15.45:6789/0 conn(0x7fe44813f650 legacy=0x7fe4480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe448146340 con 0x7fe4480a9d70 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.702+0000 7fe4477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe43c001690 con 0x7fe4480a9d70 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe4477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe4477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe4477fe640 10 monclient: dump: 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.711 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe44eae2640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.703+0000 7fe44eae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c0017f0 con 0x7fe4480a9d70 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 1 -- 172.21.15.45:0/1402053114 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe43c001c00 con 0x7fe4480a9d70 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:11.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe4477fe640 10 monclient: dump: 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe44eae2640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.704+0000 7fe44eae2640 4 set_mon_vals no callback set 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.705+0000 7fe44eae2640 10 monclient: shutdown 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.705+0000 7fe44eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:11.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.705+0000 7fe44eae2640 1 -- 172.21.15.45:0/1402053114 >> v1:172.21.15.190:6789/0 conn(0x7fe4480a9d70 legacy=0x7fe44814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.705+0000 7fe44eae2640 1 -- 172.21.15.45:0/1402053114 shutdown_connections 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.705+0000 7fe44eae2640 1 -- 172.21.15.45:0/1402053114 >> 172.21.15.45:0/1402053114 conn(0x7fe448066a00 msgr2=0x7fe448066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.706+0000 7fe44eae2640 1 -- 172.21.15.45:0/1402053114 shutdown_connections 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.706+0000 7fe44eae2640 1 -- 172.21.15.45:0/1402053114 wait complete. 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.707+0000 7fe44eae2640 10 monclient: build_initial_monmap 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.707+0000 7fe44eae2640 10 monclient: monmap: 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:11.714 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:11.715 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:11.715 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.707+0000 7fe44eae2640 1 librados: starting msgr at 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.707+0000 7fe44eae2640 1 librados: starting objecter 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.707+0000 7fe44eae2640 1 Processor -- start 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 -- start start 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 librados: setting wanted keys 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 librados: calling monclient init 2024-03-29T11:34:11.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient: init 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4480a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-c con 0x7fe44813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4481457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe448140280 con 0x7fe4481457f0 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4481ccf20 con 0x7fe4480a9d70 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44813fda0 con 0x7fe44813f650 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:11.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.708+0000 7fe44eae2640 10 monclient(hunting): authenticate will time out at 907.948069s 2024-03-29T11:34:11.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.714+0000 7fe44c87e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe4480a9d70 0x7fe448140710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:55569/0 (socket says 172.21.15.45:55569) 2024-03-29T11:34:11.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.714+0000 7fe44c87e640 1 -- 172.21.15.45:0/1442472681 learned_addr learned my addr 172.21.15.45:0/1442472681 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 client.?.objecter ms_handle_connect 0x7fe4480a9d70 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033934373 0 0) 0x7fe4481ccf20 con 0x7fe4480a9d70 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.718+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe434004eb0 con 0x7fe4480a9d70 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816041741 0 0) 0x7fe434004eb0 con 0x7fe4480a9d70 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4481ccf20 con 0x7fe4480a9d70 2024-03-29T11:34:11.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe440001cf0 con 0x7fe4480a9d70 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient(hunting): dump: 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 client.?.objecter ms_handle_connect 0x7fe4481457f0 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4015588842 0 0) 0x7fe4481ccf20 con 0x7fe4480a9d70 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 >> v1:172.21.15.45:6790/0 conn(0x7fe44813f650 legacy=0x7fe448140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 >> v1:172.21.15.45:6789/0 conn(0x7fe4481457f0 legacy=0x7fe4481e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 monclient: found mon.b 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44813ff30 con 0x7fe4480a9d70 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:11.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe440002010 con 0x7fe4480a9d70 2024-03-29T11:34:11.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe44eae2640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:34:11.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe44eae2640 10 monclient: _renew_subs 2024-03-29T11:34:11.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe44eae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:11.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4481cb340 con 0x7fe4480a9d70 2024-03-29T11:34:11.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe440001cf0 con 0x7fe4480a9d70 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.719+0000 7fe4467fc640 4 set_mon_vals no callback set 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.721+0000 7fe4457fa640 10 monclient: dump: 2024-03-29T11:34:11.748 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.720+0000 7fe44eae2640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.721+0000 7fe44eae2640 10 monclient: _renew_subs 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.721+0000 7fe44eae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.721+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4481428b0 con 0x7fe4480a9d70 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.721+0000 7fe44eae2640 1 librados: init done 2024-03-29T11:34:11.749 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe440003cc0 con 0x7fe4480a9d70 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe440084c80 con 0x7fe4480a9d70 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 10 client.4279.objecter ms_dispatch 0x7fe448000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.724+0000 7fe4457fa640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe44eae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe44eae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe44eae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe448075f10 con 0x7fe4480a9d70 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe447fff640 10 monclient: get_auth_request con 0x7fe43406de20 auth_method 0 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.725+0000 7fe4457fa640 10 client.4279.objecter ms_handle_connect 0x7fe43406de20 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.733+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4400505c0 con 0x7fe4480a9d70 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.733+0000 7fe4457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:11.750 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:11.733+0000 7fe4457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:12.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.062+0000 7fe44eae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:12.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.062+0000 7fe44eae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:12.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.062+0000 7fe44eae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:12.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.062+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4480b3680 con 0x7fe4480a9d70 2024-03-29T11:34:12.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.063+0000 7fe4457fa640 1 -- 172.21.15.45:0/1442472681 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe4400505c0 con 0x7fe4480a9d70 2024-03-29T11:34:12.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.063+0000 7fe4457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:12.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.063+0000 7fe4457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:12.068 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:12.068 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:12.071 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.067+0000 7fe44eae2640 10 librados: watch_flush enter 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.067+0000 7fe44eae2640 10 librados: watch_flush exit 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.067+0000 7fe44eae2640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.067+0000 7fe44eae2640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe43406de20 legacy=0x7fe4340702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 10 monclient: shutdown 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 >> v1:172.21.15.190:6789/0 conn(0x7fe4480a9d70 legacy=0x7fe448140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 shutdown_connections 2024-03-29T11:34:12.075 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.068+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 >> 172.21.15.45:0/1442472681 conn(0x7fe448066a00 msgr2=0x7fe4480b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:12.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.070+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 shutdown_connections 2024-03-29T11:34:12.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.070+0000 7fe44eae2640 1 -- 172.21.15.45:0/1442472681 wait complete. 2024-03-29T11:34:12.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:12.070+0000 7fe44eae2640 1 librados: shutdown 2024-03-29T11:34:12.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:18.904 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:19.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.063+0000 7f1501e7d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:19.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.063+0000 7f1501e7d640 10 monclient: build_initial_monmap 2024-03-29T11:34:19.060 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.063+0000 7f1501e7d640 10 monclient: monmap: 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:19.064507+0000 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:19.064507+0000 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:19.061 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.067+0000 7f1501e7d640 1 Processor -- start 2024-03-29T11:34:19.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.068+0000 7f1501e7d640 1 -- start start 2024-03-29T11:34:19.065 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.068+0000 7f1501e7d640 10 monclient: init 2024-03-29T11:34:19.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:19.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:19.067 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f14fc13f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc0aaef0 con 0x7f14fc13f650 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc145bd0 con 0x7f14fc1457f0 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc145eb0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:19.068 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.069+0000 7f1501e7d640 10 monclient(hunting): authenticate will time out at 915.309073s 2024-03-29T11:34:19.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fbfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f14fc0a9d70 0x7f14fc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49695/0 (socket says 172.21.15.45:49695) 2024-03-29T11:34:19.070 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fbfff640 1 -- 172.21.15.45:0/3626328388 learned_addr learned my addr 172.21.15.45:0/3626328388 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045849571 0 0) 0x7f14fc145eb0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.073+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e4003520 con 0x7f14fc0a9d70 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833902341 0 0) 0x7f14e4003520 con 0x7f14fc0a9d70 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 >> v1:172.21.15.190:6789/0 conn(0x7f14fc1457f0 legacy=0x7f14fc13ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 >> v1:172.21.15.45:6789/0 conn(0x7f14fc13f650 legacy=0x7f14fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:34:19.076 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc146340 con 0x7f14fc0a9d70 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f1501e7d640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f1501e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f14f4001a10 con 0x7f14fc0a9d70 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: dump: 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:19.077 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f1501e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 1 -- 172.21.15.45:0/3626328388 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14f4001ca0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f14fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f1501e7d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.074+0000 7f1501e7d640 4 set_mon_vals no callback set 2024-03-29T11:34:19.078 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.075+0000 7f1501e7d640 10 monclient: shutdown 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.075+0000 7f1501e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.075+0000 7f1501e7d640 1 -- 172.21.15.45:0/3626328388 >> v1:172.21.15.45:6790/0 conn(0x7f14fc0a9d70 legacy=0x7f14fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.076+0000 7f1501e7d640 1 -- 172.21.15.45:0/3626328388 shutdown_connections 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.076+0000 7f1501e7d640 1 -- 172.21.15.45:0/3626328388 >> 172.21.15.45:0/3626328388 conn(0x7f14fc066a00 msgr2=0x7f14fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.078+0000 7f1501e7d640 1 -- 172.21.15.45:0/3626328388 shutdown_connections 2024-03-29T11:34:19.079 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.078+0000 7f1501e7d640 1 -- 172.21.15.45:0/3626328388 wait complete. 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.079+0000 7f1501e7d640 10 monclient: build_initial_monmap 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.079+0000 7f1501e7d640 10 monclient: monmap: 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:19.081 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.079+0000 7f1501e7d640 1 librados: starting msgr at 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.079+0000 7f1501e7d640 1 librados: starting objecter 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.079+0000 7f1501e7d640 1 Processor -- start 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 1 -- start start 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 1 librados: setting wanted keys 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 1 librados: calling monclient init 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 10 monclient: init 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.080+0000 7f1501e7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f14fc0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:19.082 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc142900 con 0x7f14fc0a9d70 2024-03-29T11:34:19.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc141b40 con 0x7f14fc1457f0 2024-03-29T11:34:19.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc141cd0 con 0x7f14fc13f650 2024-03-29T11:34:19.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:19.083 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.081+0000 7f1501e7d640 10 monclient(hunting): authenticate will time out at 915.320179s 2024-03-29T11:34:19.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.087+0000 7f14fb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f14fc0a9d70 0x7f14fc141ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34685/0 (socket says 172.21.15.45:34685) 2024-03-29T11:34:19.085 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14fb7fe640 1 -- 172.21.15.45:0/3673266948 learned_addr learned my addr 172.21.15.45:0/3673266948 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 client.?.objecter ms_handle_connect 0x7f14fc1457f0 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 client.?.objecter ms_handle_connect 0x7f14fc0a9d70 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159569827 0 0) 0x7f14fc141b40 con 0x7f14fc1457f0 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:34:19.087 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8004eb0 con 0x7f14fc1457f0 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259644886 0 0) 0x7f14fc142900 con 0x7f14fc0a9d70 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.088+0000 7f14e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14fc141b40 con 0x7f14fc0a9d70 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 10 client.?.objecter ms_handle_connect 0x7f14fc13f650 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2783176710 0 0) 0x7f14e8004eb0 con 0x7f14fc1457f0 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.089+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14fc142900 con 0x7f14fc1457f0 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 343038544 0 0) 0x7f14fc141b40 con 0x7f14fc0a9d70 2024-03-29T11:34:19.088 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14e8004eb0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363735011 0 0) 0x7f14fc141cd0 con 0x7f14fc13f650 2024-03-29T11:34:19.090 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14fc141b40 con 0x7f14fc13f650 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f14f4004cc0 con 0x7f14fc1457f0 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.091 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f14ec001640 con 0x7f14fc0a9d70 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.092 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36205505 0 0) 0x7f14e8004eb0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 >> v1:172.21.15.45:6790/0 conn(0x7f14fc13f650 legacy=0x7f14fc1e0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 >> v1:172.21.15.190:6789/0 conn(0x7f14fc1457f0 legacy=0x7f14fc1e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 monclient: found mon.a 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:19.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc139c70 con 0x7f14fc0a9d70 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f14e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f1501e7d640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f1501e7d640 10 monclient: _renew_subs 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f1501e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.090+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc139e60 con 0x7f14fc0a9d70 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.091+0000 7f1501e7d640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.091+0000 7f1501e7d640 10 monclient: _renew_subs 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.091+0000 7f1501e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.091+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc1e1720 con 0x7f14fc0a9d70 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.091+0000 7f1501e7d640 1 librados: init done 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ec0017a0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.094 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f14ec004860 con 0x7f14fc0a9d70 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14e3fff640 10 monclient: dump: 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:19.095 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:19.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.092+0000 7f14f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:19.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.093+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f14ec0049c0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.093+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f14ec0848a0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.093+0000 7f14e3fff640 10 client.4317.objecter ms_dispatch 0x7f14fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.094+0000 7f14e3fff640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.094+0000 7f14e3fff640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.094+0000 7f14e3fff640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.096+0000 7f14faffd640 10 monclient: get_auth_request con 0x7f14e806e120 auth_method 0 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.097+0000 7f1501e7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.097+0000 7f1501e7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.097+0000 7f1501e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.097+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc005080 con 0x7f14fc0a9d70 2024-03-29T11:34:19.097 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.097+0000 7f14e3fff640 10 client.4317.objecter ms_handle_connect 0x7f14e806e120 2024-03-29T11:34:19.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.105+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14ec04ffb0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.105+0000 7f14e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:19.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.105+0000 7f14e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:19.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.470+0000 7f1501e7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:19.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.470+0000 7f1501e7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:19.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.470+0000 7f1501e7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:34:19.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.470+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14dc005330 con 0x7f14fc0a9d70 2024-03-29T11:34:19.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.471+0000 7f14e3fff640 1 -- 172.21.15.45:0/3673266948 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f14ec0520b0 con 0x7f14fc0a9d70 2024-03-29T11:34:19.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.471+0000 7f14e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:19.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.471+0000 7f14e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:19.468 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:19.468 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:19.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 10 librados: watch_flush enter 2024-03-29T11:34:19.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 10 librados: watch_flush exit 2024-03-29T11:34:19.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:34:19.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 >> v1:172.21.15.190:6816/1411598462 conn(0x7f14e806e120 legacy=0x7f14e80705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 10 monclient: shutdown 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 >> v1:172.21.15.45:6789/0 conn(0x7f14fc0a9d70 legacy=0x7f14fc141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.474+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 shutdown_connections 2024-03-29T11:34:19.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.475+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 >> 172.21.15.45:0/3673266948 conn(0x7f14fc066a00 msgr2=0x7f14fc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:19.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.475+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 shutdown_connections 2024-03-29T11:34:19.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.475+0000 7f1501e7d640 1 -- 172.21.15.45:0/3673266948 wait complete. 2024-03-29T11:34:19.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:19.475+0000 7f1501e7d640 1 librados: shutdown 2024-03-29T11:34:19.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:26.294 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:26.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.445+0000 7fa522831640 10 monclient: get_monmap_and_config 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.445+0000 7fa522831640 10 monclient: build_initial_monmap 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.445+0000 7fa522831640 10 monclient: monmap: 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:26.446497+0000 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:26.446497+0000 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:26.444 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.448+0000 7fa522831640 1 Processor -- start 2024-03-29T11:34:26.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.449+0000 7fa522831640 1 -- start start 2024-03-29T11:34:26.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.449+0000 7fa522831640 10 monclient: init 2024-03-29T11:34:26.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:26.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:26.452 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient(hunting): picked mon.noname-a con 0x7fa51c0acfa0 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient(hunting): picked mon.noname-b con 0x7fa51c072f10 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient(hunting): picked mon.noname-c con 0x7fa51c0ad380 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c0aa9f0 con 0x7fa51c0acfa0 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c0ac4c0 con 0x7fa51c0ad380 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c0ac7a0 con 0x7fa51c072f10 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.450+0000 7fa522831640 10 monclient(hunting): authenticate will time out at 922.689585s 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.451+0000 7fa52102e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa51c072f10 0x7fa51c141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39940/0 (socket says 172.21.15.45:39940) 2024-03-29T11:34:26.453 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.451+0000 7fa52102e640 1 -- 172.21.15.45:0/3866581110 learned_addr learned my addr 172.21.15.45:0/3866581110 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:26.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.455+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459901155 0 0) 0x7fa51c0ac7a0 con 0x7fa51c072f10 2024-03-29T11:34:26.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _negotiate 2024-03-29T11:34:26.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:26.456 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa508003520 con 0x7fa51c072f10 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331339914 0 0) 0x7fa51c0aa9f0 con 0x7fa51c0acfa0 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _negotiate 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c0ac7a0 con 0x7fa51c0acfa0 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753622676 0 0) 0x7fa508003520 con 0x7fa51c072f10 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 >> v1:172.21.15.190:6789/0 conn(0x7fa51c0ad380 legacy=0x7fa51c1497f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 >> v1:172.21.15.45:6789/0 conn(0x7fa51c0acfa0 legacy=0x7fa51c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: _finish_hunting 0 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 monclient: found mon.noname-b 2024-03-29T11:34:26.457 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0ab5a0 con 0x7fa51c072f10 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: _finish_auth 0 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: _check_auth_tickets 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa514001a10 con 0x7fa51c072f10 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.456+0000 7fa52082d640 10 monclient: dump: 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.458 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.457+0000 7fa522831640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.457+0000 7fa522831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.457+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa514001b70 con 0x7fa51c072f10 2024-03-29T11:34:26.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.457+0000 7fa52082d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa522831640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa522831640 4 set_mon_vals no callback set 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa52082d640 1 -- 172.21.15.45:0/3866581110 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa514001f80 con 0x7fa51c072f10 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa52082d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa52082d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.458+0000 7fa52082d640 10 monclient: dump: 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.460 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.459+0000 7fa522831640 10 monclient: shutdown 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.459+0000 7fa522831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.459+0000 7fa522831640 1 -- 172.21.15.45:0/3866581110 >> v1:172.21.15.45:6790/0 conn(0x7fa51c072f10 legacy=0x7fa51c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.462+0000 7fa522831640 1 -- 172.21.15.45:0/3866581110 shutdown_connections 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.462+0000 7fa522831640 1 -- 172.21.15.45:0/3866581110 >> 172.21.15.45:0/3866581110 conn(0x7fa51c0b3a30 msgr2=0x7fa51c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:26.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.463+0000 7fa522831640 1 -- 172.21.15.45:0/3866581110 shutdown_connections 2024-03-29T11:34:26.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.465+0000 7fa522831640 1 -- 172.21.15.45:0/3866581110 wait complete. 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.465+0000 7fa522831640 10 monclient: build_initial_monmap 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.465+0000 7fa522831640 10 monclient: monmap: 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 librados: starting msgr at 2024-03-29T11:34:26.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 librados: starting objecter 2024-03-29T11:34:26.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 Processor -- start 2024-03-29T11:34:26.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 -- start start 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 librados: setting wanted keys 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 1 librados: calling monclient init 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.466+0000 7fa522831640 10 monclient: init 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient(hunting): picked mon.noname-a con 0x7fa51c072f10 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient(hunting): picked mon.noname-c con 0x7fa51c0acfa0 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient(hunting): picked mon.noname-b con 0x7fa51c0ad380 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:26.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c12f930 con 0x7fa51c072f10 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c1df2e0 con 0x7fa51c0ad380 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c1df470 con 0x7fa51c0acfa0 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.467+0000 7fa522831640 10 monclient(hunting): authenticate will time out at 922.706624s 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.470+0000 7fa522030640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa51c0ad380 0x7fa51c1e5ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:37055/0 (socket says 172.21.15.45:37055) 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.470+0000 7fa522030640 1 -- 172.21.15.45:0/1302399626 learned_addr learned my addr 172.21.15.45:0/1302399626 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 client.?.objecter ms_handle_connect 0x7fa51c0ad380 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 client.?.objecter ms_handle_connect 0x7fa51c072f10 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535294612 0 0) 0x7fa51c1df2e0 con 0x7fa51c0ad380 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:26.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500004eb0 con 0x7fa51c0ad380 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381354689 0 0) 0x7fa51c12f930 con 0x7fa51c072f10 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.471+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c1df2e0 con 0x7fa51c072f10 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2468967414 0 0) 0x7fa500004eb0 con 0x7fa51c0ad380 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa51c12f930 con 0x7fa51c0ad380 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa518001750 con 0x7fa51c0ad380 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:26.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): dump: 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3268325682 0 0) 0x7fa51c1df2e0 con 0x7fa51c072f10 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa500004eb0 con 0x7fa51c072f10 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa50c001930 con 0x7fa51c072f10 2024-03-29T11:34:26.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient(hunting): dump: 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2659354673 0 0) 0x7fa51c12f930 con 0x7fa51c0ad380 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 >> v1:172.21.15.45:6790/0 conn(0x7fa51c0acfa0 legacy=0x7fa51c1db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 >> v1:172.21.15.45:6789/0 conn(0x7fa51c072f10 legacy=0x7fa51c1debd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 monclient: found mon.b 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c1dbe00 con 0x7fa51c0ad380 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.472+0000 7fa5127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 10 monclient: _renew_subs 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c1dc0b0 con 0x7fa51c0ad380 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 10 monclient: _renew_subs 2024-03-29T11:34:26.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:26.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c14a930 con 0x7fa51c0ad380 2024-03-29T11:34:26.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.473+0000 7fa522831640 1 librados: init done 2024-03-29T11:34:26.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.474+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa518001c60 con 0x7fa51c0ad380 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.474+0000 7fa5127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.474+0000 7fa513fff640 4 set_mon_vals no callback set 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa522831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa518004a00 con 0x7fa51c0ad380 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa5127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa5127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa5127fc640 10 monclient: dump: 2024-03-29T11:34:26.481 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.475+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fa51801a9d0 con 0x7fa51c0ad380 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.476+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fa518084960 con 0x7fa51c0ad380 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.476+0000 7fa5127fc640 10 client.4285.objecter ms_dispatch 0x7fa51c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:26.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.476+0000 7fa5127fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.476+0000 7fa5127fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.476+0000 7fa5127fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.477+0000 7fa52182f640 10 monclient: get_auth_request con 0x7fa50006de80 auth_method 0 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.477+0000 7fa522831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.477+0000 7fa522831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.477+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa504005080 con 0x7fa51c0ad380 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.477+0000 7fa5127fc640 10 client.4285.objecter ms_handle_connect 0x7fa50006de80 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.483+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5180502c0 con 0x7fa51c0ad380 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.483+0000 7fa5127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:26.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.483+0000 7fa5127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.876+0000 7fa522831640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.876+0000 7fa522831640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.876+0000 7fa522831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.876+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa504005330 con 0x7fa51c0ad380 2024-03-29T11:34:26.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.878+0000 7fa5127fc640 1 -- 172.21.15.45:0/1302399626 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fa5180521f0 con 0x7fa51c0ad380 2024-03-29T11:34:26.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.878+0000 7fa5127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:26.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.878+0000 7fa5127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:26.876 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:26.876 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:26.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 10 librados: watch_flush enter 2024-03-29T11:34:26.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 10 librados: watch_flush exit 2024-03-29T11:34:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:34:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:34:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 >> v1:172.21.15.190:6816/1411598462 conn(0x7fa50006de80 legacy=0x7fa500070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 10 monclient: shutdown 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.882+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 >> v1:172.21.15.190:6789/0 conn(0x7fa51c0ad380 legacy=0x7fa51c1e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.883+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 shutdown_connections 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.883+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 >> 172.21.15.45:0/1302399626 conn(0x7fa51c0b3a30 msgr2=0x7fa51c0b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.883+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 shutdown_connections 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.883+0000 7fa522831640 1 -- 172.21.15.45:0/1302399626 wait complete. 2024-03-29T11:34:26.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:26.883+0000 7fa522831640 1 librados: shutdown 2024-03-29T11:34:26.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:33.705 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.859+0000 7fe4c0182640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.859+0000 7fe4c0182640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.859+0000 7fe4c0182640 10 monclient: monmap: 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:33.860634+0000 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:33.860634+0000 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:33.857 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:33.858 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.860+0000 7fe4c0182640 1 Processor -- start 2024-03-29T11:34:33.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.860+0000 7fe4c0182640 1 -- start start 2024-03-29T11:34:33.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.860+0000 7fe4c0182640 10 monclient: init 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b81457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80aaef0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8145bd0 con 0x7fe4b813f650 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8145eb0 con 0x7fe4b81457f0 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.861+0000 7fe4c0182640 10 monclient(hunting): authenticate will time out at 930.100830s 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bd71d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe4b813f650 0x7fe4b80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:38372/0 (socket says 172.21.15.45:38372) 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bd71d640 1 -- 172.21.15.45:0/3919950549 learned_addr learned my addr 172.21.15.45:0/3919950549 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427615722 0 0) 0x7fe4b80aaef0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4b80a9d70 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824614388 0 0) 0x7fe4b8145eb0 con 0x7fe4b81457f0 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.862+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80aaef0 con 0x7fe4b81457f0 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162242139 0 0) 0x7fe4a4003520 con 0x7fe4b80a9d70 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 >> v1:172.21.15.45:6790/0 conn(0x7fe4b81457f0 legacy=0x7fe4b813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 >> v1:172.21.15.190:6789/0 conn(0x7fe4b813f650 legacy=0x7fe4b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 monclient: found mon.noname-a 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b8146340 con 0x7fe4b80a9d70 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe4b0001940 con 0x7fe4b80a9d70 2024-03-29T11:34:33.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: dump: 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4c0182640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4c0182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0001aa0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4c0182640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4c0182640 4 set_mon_vals no callback set 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 1 -- 172.21.15.45:0/3919950549 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe4b0001ef0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.863+0000 7fe4bcf1c640 10 monclient: dump: 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:33.865 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 10 monclient: shutdown 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 1 -- 172.21.15.45:0/3919950549 >> v1:172.21.15.45:6789/0 conn(0x7fe4b80a9d70 legacy=0x7fe4b814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 1 -- 172.21.15.45:0/3919950549 shutdown_connections 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 1 -- 172.21.15.45:0/3919950549 >> 172.21.15.45:0/3919950549 conn(0x7fe4b8066a00 msgr2=0x7fe4b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 1 -- 172.21.15.45:0/3919950549 shutdown_connections 2024-03-29T11:34:33.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.864+0000 7fe4c0182640 1 -- 172.21.15.45:0/3919950549 wait complete. 2024-03-29T11:34:33.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.865+0000 7fe4c0182640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.865+0000 7fe4c0182640 10 monclient: monmap: 2024-03-29T11:34:33.871 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:33.871 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 librados: starting msgr at 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 librados: starting objecter 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 Processor -- start 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 -- start start 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 librados: setting wanted keys 2024-03-29T11:34:33.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 1 librados: calling monclient init 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.866+0000 7fe4c0182640 10 monclient: init 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b81457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8140210 con 0x7fe4b80a9d70 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b81da820 con 0x7fe4b81457f0 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b813fda0 con 0x7fe4b813f650 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.867+0000 7fe4c0182640 10 monclient(hunting): authenticate will time out at 930.107064s 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4bd71d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe4b813f650 0x7fe4b81424d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37986/0 (socket says 172.21.15.45:37986) 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4bd71d640 1 -- 172.21.15.45:0/593790248 learned_addr learned my addr 172.21.15.45:0/593790248 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe4b813f650 2024-03-29T11:34:33.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe4b80a9d70 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955520414 0 0) 0x7fe4b8140210 con 0x7fe4b80a9d70 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.868+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866689816 0 0) 0x7fe4b813fda0 con 0x7fe4b813f650 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:34:33.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8140210 con 0x7fe4b813f650 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268468167 0 0) 0x7fe4b8140210 con 0x7fe4b813f650 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b813fda0 con 0x7fe4b813f650 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe4a80019b0 con 0x7fe4b813f650 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:33.875 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 592145223 0 0) 0x7fe49c004eb0 con 0x7fe4b80a9d70 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.869+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b8140210 con 0x7fe4b80a9d70 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe4b4001850 con 0x7fe4b80a9d70 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:33.876 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1642575166 0 0) 0x7fe4b813fda0 con 0x7fe4b813f650 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 >> v1:172.21.15.190:6789/0 conn(0x7fe4b81457f0 legacy=0x7fe4b81dad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 >> v1:172.21.15.45:6789/0 conn(0x7fe4b80a9d70 legacy=0x7fe4b813e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 monclient: found mon.c 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b813ff30 con 0x7fe4b813f650 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 10 monclient: _renew_subs 2024-03-29T11:34:33.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4a80047e0 con 0x7fe4b813f650 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b8142be0 con 0x7fe4b813f650 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe4a8004af0 con 0x7fe4b813f650 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4affff640 4 set_mon_vals no callback set 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 10 monclient: _renew_subs 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.870+0000 7fe4c0182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.871+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b81db630 con 0x7fe4b813f650 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.871+0000 7fe4c0182640 1 librados: init done 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.871+0000 7fe4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.871+0000 7fe4ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.871+0000 7fe4ae7fc640 10 monclient: dump: 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.872+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe4a801aac0 con 0x7fe4b813f650 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe4a8084b10 con 0x7fe4b813f650 2024-03-29T11:34:33.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4ae7fc640 10 client.4322.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4ae7fc640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4ae7fc640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4ae7fc640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4bdf1e640 10 monclient: get_auth_request con 0x7fe49c06ded0 auth_method 0 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4c0182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4c0182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4c0182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.873+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b81d4f00 con 0x7fe4b813f650 2024-03-29T11:34:33.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.874+0000 7fe4ae7fc640 10 client.4322.objecter ms_handle_connect 0x7fe49c06ded0 2024-03-29T11:34:33.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.884+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b81d4f00 con 0x7fe4b813f650 2024-03-29T11:34:33.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.884+0000 7fe4ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:33.884+0000 7fe4ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:34.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.287+0000 7fe4c0182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.287+0000 7fe4c0182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.287+0000 7fe4c0182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.287+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b81d4f00 con 0x7fe4b813f650 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.288+0000 7fe4ae7fc640 1 -- 172.21.15.45:0/593790248 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe4b81d4f00 con 0x7fe4b813f650 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.288+0000 7fe4ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:34.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.288+0000 7fe4ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:34.286 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:34.286 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:34.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 10 librados: watch_flush enter 2024-03-29T11:34:34.288 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 10 librados: watch_flush exit 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe49c06ded0 legacy=0x7fe49c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 10 monclient: shutdown 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 >> v1:172.21.15.45:6790/0 conn(0x7fe4b813f650 legacy=0x7fe4b81424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 shutdown_connections 2024-03-29T11:34:34.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 >> 172.21.15.45:0/593790248 conn(0x7fe4b8066a00 msgr2=0x7fe4b813b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.291+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 shutdown_connections 2024-03-29T11:34:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.292+0000 7fe4c0182640 1 -- 172.21.15.45:0/593790248 wait complete. 2024-03-29T11:34:34.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:34.292+0000 7fe4c0182640 1 librados: shutdown 2024-03-29T11:34:34.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:41.114 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.249+0000 7f2b2bc0f640 10 monclient: get_monmap_and_config 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.249+0000 7f2b2bc0f640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.249+0000 7f2b2bc0f640 10 monclient: monmap: 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:41.250681+0000 2024-03-29T11:34:41.247 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:41.250681+0000 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.250+0000 7f2b2bc0f640 1 Processor -- start 2024-03-29T11:34:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.250+0000 7f2b2bc0f640 1 -- start start 2024-03-29T11:34:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.250+0000 7f2b2bc0f640 10 monclient: init 2024-03-29T11:34:41.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b24070170 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b240713f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b24070550 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2413dbd0 con 0x7f2b24070550 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2413ff70 con 0x7f2b240713f0 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b24140250 con 0x7f2b24070170 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b2bc0f640 10 monclient(hunting): authenticate will time out at 937.491492s 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b299ab640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2b24070170 0x7f2b2413d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53600/0 (socket says 172.21.15.45:53600) 2024-03-29T11:34:41.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b299ab640 1 -- 172.21.15.45:0/3190622164 learned_addr learned my addr 172.21.15.45:0/3190622164 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290434182 0 0) 0x7f2b24140250 con 0x7f2b24070170 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.252+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b10003520 con 0x7f2b24070170 2024-03-29T11:34:41.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80890469 0 0) 0x7f2b2413ff70 con 0x7f2b240713f0 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b24140250 con 0x7f2b240713f0 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 596953499 0 0) 0x7f2b10003520 con 0x7f2b24070170 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 >> v1:172.21.15.190:6789/0 conn(0x7f2b240713f0 legacy=0x7f2b241406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.253+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 >> v1:172.21.15.45:6789/0 conn(0x7f2b24070550 legacy=0x7f2b2414b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:41.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:41.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 1 monclient: found mon.noname-b 2024-03-29T11:34:41.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:41.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24140df0 con 0x7f2b24070170 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: _finish_auth 0 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2b20001690 con 0x7f2b24070170 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: dump: 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.254 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b2bc0f640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b2bc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b200017f0 con 0x7f2b24070170 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:41.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 1 -- 172.21.15.45:0/3190622164 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2b20001bc0 con 0x7f2b24070170 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b2bc0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b2bc0f640 4 set_mon_vals no callback set 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.255+0000 7f2b289a9640 10 monclient: dump: 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:41.256 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:41.257 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:41.257 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.256+0000 7f2b2bc0f640 10 monclient: shutdown 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.256+0000 7f2b2bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.256+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/3190622164 >> v1:172.21.15.45:6790/0 conn(0x7f2b24070170 legacy=0x7f2b2413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.256+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/3190622164 shutdown_connections 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.257+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/3190622164 >> 172.21.15.45:0/3190622164 conn(0x7f2b24067610 msgr2=0x7f2b240b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.257+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/3190622164 shutdown_connections 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.257+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/3190622164 wait complete. 2024-03-29T11:34:41.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 10 monclient: build_initial_monmap 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 10 monclient: monmap: 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 librados: starting msgr at 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 librados: starting objecter 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 Processor -- start 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 -- start start 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 librados: setting wanted keys 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 1 librados: calling monclient init 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.258+0000 7f2b2bc0f640 10 monclient: init 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.259+0000 7f2b2bc0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.259+0000 7f2b2bc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.259+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b24070170 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.259+0000 7f2b299ab640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2b24070170 0x7f2b2413f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:33396/0 (socket says 172.21.15.45:33396) 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b24070550 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:41.268 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b240713f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b299ab640 1 -- 172.21.15.45:0/1927912715 learned_addr learned my addr 172.21.15.45:0/1927912715 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2413b160 con 0x7f2b24070550 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2413e570 con 0x7f2b24070170 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2413f760 con 0x7f2b240713f0 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b2bc0f640 10 monclient(hunting): authenticate will time out at 937.499475s 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f2b24070170 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f2b240713f0 2024-03-29T11:34:41.269 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081705496 0 0) 0x7f2b2413e570 con 0x7f2b24070170 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08004eb0 con 0x7f2b24070170 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137956999 0 0) 0x7f2b2413f760 con 0x7f2b240713f0 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:34:41.270 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.260+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b2413e570 con 0x7f2b240713f0 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520253258 0 0) 0x7f2b08004eb0 con 0x7f2b24070170 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b2413f760 con 0x7f2b24070170 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2b20001c40 con 0x7f2b24070170 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.271 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475962230 0 0) 0x7f2b2413e570 con 0x7f2b240713f0 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b08004eb0 con 0x7f2b240713f0 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2b1c001ab0 con 0x7f2b240713f0 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:41.272 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3183827733 0 0) 0x7f2b08004eb0 con 0x7f2b240713f0 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 >> v1:172.21.15.190:6789/0 conn(0x7f2b24070170 legacy=0x7f2b2413f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.261+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 >> v1:172.21.15.45:6789/0 conn(0x7f2b24070550 legacy=0x7f2b2413b680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.262+0000 7f2b1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.262+0000 7f2b1a7fc640 1 monclient: found mon.c 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2413f8f0 con 0x7f2b240713f0 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b2bc0f640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b2bc0f640 10 monclient: _renew_subs 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1c001fc0 con 0x7f2b240713f0 2024-03-29T11:34:41.274 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b2bc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b241e6f90 con 0x7f2b240713f0 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b2bc0f640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.263+0000 7f2b1bfff640 4 set_mon_vals no callback set 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2b1c003b10 con 0x7f2b240713f0 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b1a7fc640 10 monclient: dump: 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.275 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b2bc0f640 10 monclient: _renew_subs 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b2bc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b241e7300 con 0x7f2b240713f0 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.264+0000 7f2b2bc0f640 1 librados: init done 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.265+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f2b1c019ae0 con 0x7f2b240713f0 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.265+0000 7f2b2bc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.276 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.265+0000 7f2b2bc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.265+0000 7f2b2bc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.265+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b241387c0 con 0x7f2b240713f0 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2b1c084ac0 con 0x7f2b240713f0 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b299ab640 10 monclient: get_auth_request con 0x7f2b0806ddb0 auth_method 0 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 10 client.4328.objecter ms_dispatch 0x7f2b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:34:41.277 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.266+0000 7f2b1a7fc640 10 client.4328.objecter ms_handle_connect 0x7f2b0806ddb0 2024-03-29T11:34:41.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.273+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b241387c0 con 0x7f2b240713f0 2024-03-29T11:34:41.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.273+0000 7f2b1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:41.278 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.273+0000 7f2b1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:41.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b2bc0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b2bc0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b2bc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:41.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b241387c0 con 0x7f2b240713f0 2024-03-29T11:34:41.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b1a7fc640 1 -- 172.21.15.45:0/1927912715 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f2b241387c0 con 0x7f2b240713f0 2024-03-29T11:34:41.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:41.586 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:41.586 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:41.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.586+0000 7f2b1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 10 librados: watch_flush enter 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 10 librados: watch_flush exit 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 >> v1:172.21.15.190:6816/1411598462 conn(0x7f2b0806ddb0 legacy=0x7f2b08070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 10 monclient: shutdown 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 >> v1:172.21.15.45:6790/0 conn(0x7f2b240713f0 legacy=0x7f2b2413bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 shutdown_connections 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 >> 172.21.15.45:0/1927912715 conn(0x7f2b24067610 msgr2=0x7f2b24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.590+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 shutdown_connections 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.591+0000 7f2b2bc0f640 1 -- 172.21.15.45:0/1927912715 wait complete. 2024-03-29T11:34:41.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:41.591+0000 7f2b2bc0f640 1 librados: shutdown 2024-03-29T11:34:41.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:48.410 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:48.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.549+0000 7f59f7201640 10 monclient: get_monmap_and_config 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.549+0000 7f59f7201640 10 monclient: build_initial_monmap 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.549+0000 7f59f7201640 10 monclient: monmap: 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:48.551082+0000 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:48.551082+0000 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:48.548 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.550+0000 7f59f7201640 1 Processor -- start 2024-03-29T11:34:48.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 1 -- start start 2024-03-29T11:34:48.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient: init 2024-03-29T11:34:48.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-a con 0x7f59f006e940 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-b con 0x7f59f006ed20 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-c con 0x7f59f013ec90 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f0070c00 con 0x7f59f006e940 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.551+0000 7f59f7201640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f0071290 con 0x7f59f013ec90 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f59fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f59f006ed20 0x7f59f013e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:59470/0 (socket says 172.21.15.45:59470) 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f59fe640 1 -- 172.21.15.45:0/2136374480 learned_addr learned my addr 172.21.15.45:0/2136374480 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f013f120 con 0x7f59f006ed20 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f7201640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f7201640 10 monclient(hunting): authenticate will time out at 944.791455s 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545411706 0 0) 0x7f59f013f120 con 0x7f59f006ed20 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:48.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59dc003440 con 0x7f59f006ed20 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848095254 0 0) 0x7f59f0071290 con 0x7f59f013ec90 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.552+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f013f120 con 0x7f59f013ec90 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.553+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904533885 0 0) 0x7f59f013f120 con 0x7f59f013ec90 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.553+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 >> v1:172.21.15.45:6790/0 conn(0x7f59f006ed20 legacy=0x7f59f013e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.553+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 >> v1:172.21.15.45:6789/0 conn(0x7f59f006e940 legacy=0x7f59f013b5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:48.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:48.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 1 monclient: found mon.noname-c 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f006fd80 con 0x7f59f013ec90 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 10 monclient: _finish_auth 0 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.555+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f59e80019b0 con 0x7f59f013ec90 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: dump: 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.554 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f7201640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:34:48.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f7201640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:48.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59e8001c00 con 0x7f59f013ec90 2024-03-29T11:34:48.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 1 -- 172.21.15.45:0/2136374480 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f59e8001f90 con 0x7f59f013ec90 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f51fd640 10 monclient: dump: 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:48.557 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f7201640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.556+0000 7f59f7201640 4 set_mon_vals no callback set 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.557+0000 7f59f7201640 10 monclient: shutdown 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.557+0000 7f59f7201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.557+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 >> v1:172.21.15.190:6789/0 conn(0x7f59f013ec90 legacy=0x7f59f0149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.558+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 shutdown_connections 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.558+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 >> 172.21.15.45:0/2136374480 conn(0x7f59f00b3fd0 msgr2=0x7f59f00b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.558+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 shutdown_connections 2024-03-29T11:34:48.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.558+0000 7f59f7201640 1 -- 172.21.15.45:0/2136374480 wait complete. 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.559+0000 7f59f7201640 10 monclient: build_initial_monmap 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.559+0000 7f59f7201640 10 monclient: monmap: 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:48.566 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 librados: starting msgr at 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 librados: starting objecter 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 Processor -- start 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 -- start start 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 librados: setting wanted keys 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 1 librados: calling monclient init 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.560+0000 7f59f7201640 10 monclient: init 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-a con 0x7f59f006e940 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-b con 0x7f59f006ed20 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient(hunting): picked mon.noname-c con 0x7f59f013ec90 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f01db560 con 0x7f59f006e940 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f0138060 con 0x7f59f006ed20 2024-03-29T11:34:48.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f0137680 con 0x7f59f013ec90 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f7201640 10 monclient(hunting): authenticate will time out at 944.800683s 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f59fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 0x7f59f01da740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:51960/0 (socket says 172.21.15.45:51960) 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.561+0000 7f59f59fe640 1 -- 172.21.15.45:0/49028351 learned_addr learned my addr 172.21.15.45:0/49028351 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 client.?.objecter ms_handle_connect 0x7f59f006ed20 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 client.?.objecter ms_handle_connect 0x7f59f013ec90 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234570417 0 0) 0x7f59f0138060 con 0x7f59f006ed20 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.562+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d4004eb0 con 0x7f59f006ed20 2024-03-29T11:34:48.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679577275 0 0) 0x7f59f0137680 con 0x7f59f013ec90 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.563+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0138060 con 0x7f59f013ec90 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024249000 0 0) 0x7f59d4004eb0 con 0x7f59f006ed20 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59f0137680 con 0x7f59f006ed20 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 client.?.objecter ms_handle_connect 0x7f59f006e940 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f59e0001830 con 0x7f59f006ed20 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:48.569 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560127588 0 0) 0x7f59f0138060 con 0x7f59f013ec90 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59f59fe640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 legacy=0x7f59f01da740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59f59fe640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 legacy=0x7f59f01da740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:48.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59f59fe640 1 --1- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 0x7f59f01da740 :-1 s=OPENED pgs=337597 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59f59fe640 1 --1- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 0x7f59f01da740 :-1 s=OPENED pgs=337597 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d4004eb0 con 0x7f59f013ec90 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.190:6789/0 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f59e8001f10 con 0x7f59f013ec90 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:48.571 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817306094 0 0) 0x7f59f01db560 con 0x7f59f006e940 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0138060 con 0x7f59f006e940 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.564+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506816312 0 0) 0x7f59d4004eb0 con 0x7f59f013ec90 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6789/0 conn(0x7f59f006ed20 legacy=0x7f59f01da740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.45:6789/0 conn(0x7f59f006e940 legacy=0x7f59f0135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 1 monclient: found mon.c 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:48.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f0137960 con 0x7f59f013ec90 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 10 monclient: _renew_subs 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f0136380 con 0x7f59f013ec90 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 10 monclient: _renew_subs 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f01389c0 con 0x7f59f013ec90 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59f7201640 1 librados: init done 2024-03-29T11:34:48.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.565+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59e8003c20 con 0x7f59f013ec90 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f59e8003f30 con 0x7f59f013ec90 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59e6ffd640 10 monclient: dump: 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:48.574 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.566+0000 7f59f49fc640 4 set_mon_vals no callback set 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.567+0000 7f59f7201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.567+0000 7f59f7201640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.567+0000 7f59f7201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:48.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.567+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8005080 con 0x7f59f013ec90 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f59e8019f00 con 0x7f59f013ec90 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f59e8083b80 con 0x7f59f013ec90 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 10 client.4334.objecter ms_dispatch 0x7f59f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.569+0000 7f59e6ffd640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.570+0000 7f59f61ff640 10 monclient: get_auth_request con 0x7f59d406dfd0 auth_method 0 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.570+0000 7f59e6ffd640 10 client.4334.objecter ms_handle_connect 0x7f59d406dfd0 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.576+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59e804f460 con 0x7f59f013ec90 2024-03-29T11:34:48.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.576+0000 7f59e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:48.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.576+0000 7f59e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:48.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.898+0000 7f59f7201640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.898+0000 7f59f7201640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.898+0000 7f59f7201640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:34:48.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.898+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d8005330 con 0x7f59f013ec90 2024-03-29T11:34:48.897 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:48.897 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:48.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.899+0000 7f59e6ffd640 1 -- 172.21.15.45:0/49028351 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f59e8051390 con 0x7f59f013ec90 2024-03-29T11:34:48.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.899+0000 7f59e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.899+0000 7f59e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 10 librados: watch_flush enter 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 10 librados: watch_flush exit 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.190:6816/1411598462 conn(0x7f59d406dfd0 legacy=0x7f59d4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 10 monclient: shutdown 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 >> v1:172.21.15.45:6790/0 conn(0x7f59f013ec90 legacy=0x7f59f01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 shutdown_connections 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 >> 172.21.15.45:0/49028351 conn(0x7f59f00b3fd0 msgr2=0x7f59f013b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 shutdown_connections 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.903+0000 7f59f7201640 1 -- 172.21.15.45:0/49028351 wait complete. 2024-03-29T11:34:48.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:48.904+0000 7f59f7201640 1 librados: shutdown 2024-03-29T11:34:48.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:55.724 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.873+0000 7fcc5bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.873+0000 7fcc5bfff640 10 monclient: build_initial_monmap 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.873+0000 7fcc5bfff640 10 monclient: monmap: 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:34:55.874349+0000 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:34:55.874349+0000 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.871 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:55.872 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:34:55.872 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:34:55.872 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.874+0000 7fcc5bfff640 1 Processor -- start 2024-03-29T11:34:55.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.874+0000 7fcc5bfff640 1 -- start start 2024-03-29T11:34:55.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.874+0000 7fcc5bfff640 10 monclient: init 2024-03-29T11:34:55.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.875+0000 7fcc5bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:55.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.875+0000 7fcc5bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:55.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.875+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fcc5c071f40 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:55.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.875+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fcc5c0add10 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:55.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.875+0000 7fcc5affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcc5c071f40 0x7fcc5c13f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:55092/0 (socket says 172.21.15.45:55092) 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5c13fa10 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0af080 con 0x7fcc5c13fa10 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0aad20 con 0x7fcc5c071f40 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0aeb60 con 0x7fcc5c0add10 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5affd640 1 -- 172.21.15.45:0/628325456 learned_addr learned my addr 172.21.15.45:0/628325456 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.878+0000 7fcc5bfff640 10 monclient(hunting): authenticate will time out at 952.117209s 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951933691 0 0) 0x7fcc5c0af080 con 0x7fcc5c13fa10 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48003520 con 0x7fcc5c13fa10 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477481078 0 0) 0x7fcc5c0aad20 con 0x7fcc5c071f40 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.879+0000 7fcc5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5c0af080 con 0x7fcc5c071f40 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365458060 0 0) 0x7fcc48003520 con 0x7fcc5c13fa10 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 >> v1:172.21.15.45:6790/0 conn(0x7fcc5c0add10 legacy=0x7fcc5c0a94e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 >> v1:172.21.15.190:6789/0 conn(0x7fcc5c071f40 legacy=0x7fcc5c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:55.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0af2c0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcc54001a10 con 0x7fcc5c13fa10 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.880+0000 7fcc5a7fc640 10 monclient: dump: 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.882 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.881+0000 7fcc5bfff640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:34:55.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.881+0000 7fcc5bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.882+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc54001b70 con 0x7fcc5c13fa10 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5a7fc640 1 -- 172.21.15.45:0/628325456 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcc54001fc0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5bfff640 4 set_mon_vals no callback set 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.883+0000 7fcc5a7fc640 10 monclient: dump: 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:55.886 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.884+0000 7fcc5bfff640 10 monclient: shutdown 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.884+0000 7fcc5bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.884+0000 7fcc5bfff640 1 -- 172.21.15.45:0/628325456 >> v1:172.21.15.45:6789/0 conn(0x7fcc5c13fa10 legacy=0x7fcc5c14d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.884+0000 7fcc5bfff640 1 -- 172.21.15.45:0/628325456 shutdown_connections 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.884+0000 7fcc5bfff640 1 -- 172.21.15.45:0/628325456 >> 172.21.15.45:0/628325456 conn(0x7fcc5c0673e0 msgr2=0x7fcc5c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.885+0000 7fcc5bfff640 1 -- 172.21.15.45:0/628325456 shutdown_connections 2024-03-29T11:34:55.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.885+0000 7fcc5bfff640 1 -- 172.21.15.45:0/628325456 wait complete. 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.886+0000 7fcc5bfff640 10 monclient: build_initial_monmap 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.886+0000 7fcc5bfff640 10 monclient: monmap: 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.886+0000 7fcc5bfff640 1 librados: starting msgr at 2024-03-29T11:34:55.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.886+0000 7fcc5bfff640 1 librados: starting objecter 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.886+0000 7fcc5bfff640 1 Processor -- start 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 1 -- start start 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 1 librados: setting wanted keys 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 1 librados: calling monclient init 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 10 monclient: init 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.887+0000 7fcc5bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.888+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5c071f40 addr v1:172.21.15.45:6789/0 2024-03-29T11:34:55.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.888+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fcc5c0add10 addr v1:172.21.15.45:6790/0 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.888+0000 7fcc5bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fcc5c13fa10 addr v1:172.21.15.190:6789/0 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5bfff640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c1428c0 con 0x7fcc5c071f40 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5bfff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c143f90 con 0x7fcc5c13fa10 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5bfff640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c142d50 con 0x7fcc5c0add10 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5b7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcc5c13fa10 0x7fcc5c1e28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:58300/0 (socket says 172.21.15.45:58300) 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5b7fe640 1 -- 172.21.15.45:0/3868373938 learned_addr learned my addr 172.21.15.45:0/3868373938 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.889+0000 7fcc5bfff640 10 monclient(hunting): authenticate will time out at 952.128281s 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 client.?.objecter ms_handle_connect 0x7fcc5c13fa10 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:55.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830757924 0 0) 0x7fcc5c143f90 con 0x7fcc5c13fa10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.890+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c004eb0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 client.?.objecter ms_handle_connect 0x7fcc5c0add10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 client.?.objecter ms_handle_connect 0x7fcc5c071f40 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3383497319 0 0) 0x7fcc3c004eb0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc5c143f90 con 0x7fcc5c13fa10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcc54001f40 con 0x7fcc5c13fa10 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 monclient(hunting): dump: 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.892+0000 7fcc537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482612594 0 0) 0x7fcc5c142d50 con 0x7fcc5c0add10 2024-03-29T11:34:55.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c004eb0 con 0x7fcc5c0add10 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913681858 0 0) 0x7fcc5c143f90 con 0x7fcc5c13fa10 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 >> v1:172.21.15.45:6790/0 conn(0x7fcc5c0add10 legacy=0x7fcc5c135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 >> v1:172.21.15.45:6789/0 conn(0x7fcc5c071f40 legacy=0x7fcc5c13fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 monclient: found mon.b 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0acb00 con 0x7fcc5c13fa10 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:55.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:55.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:55.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:34:55.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 10 monclient: _renew_subs 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c141b90 con 0x7fcc5c13fa10 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 10 monclient: _renew_subs 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c140760 con 0x7fcc5c13fa10 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.893+0000 7fcc5bfff640 1 librados: init done 2024-03-29T11:34:55.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc54003c20 con 0x7fcc5c13fa10 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcc54004070 con 0x7fcc5c13fa10 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc537fe640 10 monclient: dump: 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:34:55.901 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.894+0000 7fcc59ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.896+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fcc540041d0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.897+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fcc54084e00 con 0x7fcc5c13fa10 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.897+0000 7fcc537fe640 10 client.4306.objecter ms_dispatch 0x7fcc5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.897+0000 7fcc537fe640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.897+0000 7fcc537fe640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.897+0000 7fcc537fe640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.898+0000 7fcc5bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.898+0000 7fcc5bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.898+0000 7fcc5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.898+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc40005080 con 0x7fcc5c13fa10 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.901+0000 7fcc5affd640 10 monclient: get_auth_request con 0x7fcc3c06dd40 auth_method 0 2024-03-29T11:34:55.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.901+0000 7fcc537fe640 10 client.4306.objecter ms_handle_connect 0x7fcc3c06dd40 2024-03-29T11:34:55.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.907+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc540506e0 con 0x7fcc5c13fa10 2024-03-29T11:34:55.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.907+0000 7fcc537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:55.907+0000 7fcc537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.250+0000 7fcc5bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.250+0000 7fcc5bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.250+0000 7fcc5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:34:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.250+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc40005330 con 0x7fcc5c13fa10 2024-03-29T11:34:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.251+0000 7fcc537fe640 1 -- 172.21.15.45:0/3868373938 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fcc54052610 con 0x7fcc5c13fa10 2024-03-29T11:34:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.251+0000 7fcc537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.251+0000 7fcc537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:56.250 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:34:56.250 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.255+0000 7fcc5bfff640 10 librados: watch_flush enter 2024-03-29T11:34:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.256+0000 7fcc5bfff640 10 librados: watch_flush exit 2024-03-29T11:34:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.256+0000 7fcc5bfff640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:34:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.256+0000 7fcc5bfff640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:34:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 >> v1:172.21.15.190:6816/1411598462 conn(0x7fcc3c06dd40 legacy=0x7fcc3c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 10 monclient: shutdown 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 >> v1:172.21.15.190:6789/0 conn(0x7fcc5c13fa10 legacy=0x7fcc5c1e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 shutdown_connections 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 >> 172.21.15.45:0/3868373938 conn(0x7fcc5c0673e0 msgr2=0x7fcc5c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 shutdown_connections 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 -- 172.21.15.45:0/3868373938 wait complete. 2024-03-29T11:34:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:34:56.257+0000 7fcc5bfff640 1 librados: shutdown 2024-03-29T11:34:56.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:03.080 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:03.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.235+0000 7f505e2b5640 10 monclient: get_monmap_and_config 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.235+0000 7f505e2b5640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.235+0000 7f505e2b5640 10 monclient: monmap: 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:03.235843+0000 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:03.235843+0000 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:03.233 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.239+0000 7f505e2b5640 1 Processor -- start 2024-03-29T11:35:03.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.239+0000 7f505e2b5640 1 -- start start 2024-03-29T11:35:03.236 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.239+0000 7f505e2b5640 10 monclient: init 2024-03-29T11:35:03.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.240+0000 7f505e2b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.240+0000 7f505e2b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:03.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.240+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-c con 0x7f50580b0500 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:03.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.240+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-b con 0x7f50580b08e0 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-a con 0x7f50580aeca0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058140620 con 0x7f50580aeca0 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f505813daf0 con 0x7f50580b0500 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580ae180 con 0x7f50580b08e0 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.242+0000 7f505e2b5640 10 monclient(hunting): authenticate will time out at 959.480882s 2024-03-29T11:35:03.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.245+0000 7f5057fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f50580b0500 0x7f505813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:60105/0 (socket says 172.21.15.45:60105) 2024-03-29T11:35:03.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.245+0000 7f5057fff640 1 -- 172.21.15.45:0/3788699428 learned_addr learned my addr 172.21.15.45:0/3788699428 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249900102 0 0) 0x7f505813daf0 con 0x7f50580b0500 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:35:03.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5040003520 con 0x7f50580b0500 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856108653 0 0) 0x7f50580ae180 con 0x7f50580b08e0 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f505813daf0 con 0x7f50580b08e0 2024-03-29T11:35:03.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064583689 0 0) 0x7f505813daf0 con 0x7f50580b08e0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 >> v1:172.21.15.190:6789/0 conn(0x7f50580b0500 legacy=0x7f505813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 >> v1:172.21.15.45:6789/0 conn(0x7f50580aeca0 legacy=0x7f505814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 monclient: found mon.noname-b 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50580accf0 con 0x7f50580b08e0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f504c001670 con 0x7f50580b08e0 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.248+0000 7f5056ffd640 10 monclient: dump: 2024-03-29T11:35:03.247 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f505e2b5640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f505e2b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f504c001850 con 0x7f50580b08e0 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 1 -- 172.21.15.45:0/3788699428 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f504c001c60 con 0x7f50580b08e0 2024-03-29T11:35:03.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f5056ffd640 10 monclient: dump: 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f505e2b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:03.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.249+0000 7f505e2b5640 4 set_mon_vals no callback set 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.250+0000 7f505e2b5640 10 monclient: shutdown 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.250+0000 7f505e2b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.250+0000 7f505e2b5640 1 -- 172.21.15.45:0/3788699428 >> v1:172.21.15.45:6790/0 conn(0x7f50580b08e0 legacy=0x7f505813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.251+0000 7f505e2b5640 1 -- 172.21.15.45:0/3788699428 shutdown_connections 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.251+0000 7f505e2b5640 1 -- 172.21.15.45:0/3788699428 >> 172.21.15.45:0/3788699428 conn(0x7f505806b660 msgr2=0x7f505806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:03.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.253+0000 7f505e2b5640 1 -- 172.21.15.45:0/3788699428 shutdown_connections 2024-03-29T11:35:03.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.253+0000 7f505e2b5640 1 -- 172.21.15.45:0/3788699428 wait complete. 2024-03-29T11:35:03.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.254+0000 7f505e2b5640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.254+0000 7f505e2b5640 10 monclient: monmap: 2024-03-29T11:35:03.254 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.254+0000 7f505e2b5640 1 librados: starting msgr at 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.254+0000 7f505e2b5640 1 librados: starting objecter 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.255+0000 7f505e2b5640 1 Processor -- start 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.255+0000 7f505e2b5640 1 -- start start 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.255+0000 7f505e2b5640 1 librados: setting wanted keys 2024-03-29T11:35:03.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.255+0000 7f505e2b5640 1 librados: calling monclient init 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.255+0000 7f505e2b5640 10 monclient: init 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.256+0000 7f505e2b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.256+0000 7f505e2b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.256+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-b con 0x7f50580aeca0 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.256+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-c con 0x7f50580b0500 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:03.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 10 monclient(hunting): picked mon.noname-a con 0x7f50580aaae0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50581caa00 con 0x7f50580aaae0 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50581cace0 con 0x7f50580aeca0 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50581c7f10 con 0x7f50580b0500 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.259+0000 7f505e2b5640 10 monclient(hunting): authenticate will time out at 959.497855s 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5057fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f50580aeca0 0x7f50581cb470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:60217/0 (socket says 172.21.15.45:60217) 2024-03-29T11:35:03.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5057fff640 1 -- 172.21.15.45:0/3661117825 learned_addr learned my addr 172.21.15.45:0/3661117825 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 client.?.objecter ms_handle_connect 0x7f50580aeca0 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 client.?.objecter ms_handle_connect 0x7f50580b0500 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703276926 0 0) 0x7f50581c7f10 con 0x7f50580b0500 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044004eb0 con 0x7f50580b0500 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527461749 0 0) 0x7f50581cace0 con 0x7f50580aeca0 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:35:03.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50581c7f10 con 0x7f50580aeca0 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413511415 0 0) 0x7f5044004eb0 con 0x7f50580b0500 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50581cace0 con 0x7f50580b0500 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f504c001c60 con 0x7f50580b0500 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:03.260 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:03.261 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.261+0000 7f5054ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:03.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 209525969 0 0) 0x7f50581c7f10 con 0x7f50580aeca0 2024-03-29T11:35:03.261 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5044004eb0 con 0x7f50580aeca0 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5048001950 con 0x7f50580aeca0 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:03.262 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 client.?.objecter ms_handle_connect 0x7f50580aaae0 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802722537 0 0) 0x7f5044004eb0 con 0x7f50580aeca0 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 >> v1:172.21.15.45:6790/0 conn(0x7f50580b0500 legacy=0x7f50581c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 >> v1:172.21.15.45:6789/0 conn(0x7f50580aaae0 legacy=0x7f50581c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.263+0000 7f5054ff9640 1 monclient: found mon.b 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505813c310 con 0x7f50580aeca0 2024-03-29T11:35:03.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f5054ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 10 monclient: _renew_subs 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:03.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581c9b20 con 0x7f50580aeca0 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 10 monclient: _renew_subs 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581c9680 con 0x7f50580aeca0 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.264+0000 7f505e2b5640 1 librados: init done 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5048001d60 con 0x7f50580aeca0 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f50480049c0 con 0x7f50580aeca0 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.265+0000 7f5054ff9640 10 monclient: dump: 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.265 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.266+0000 7f505e2b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.266+0000 7f505e2b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.266+0000 7f505e2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.266+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50580ad0c0 con 0x7f50580aeca0 2024-03-29T11:35:03.266 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.266+0000 7f50567fc640 4 set_mon_vals no callback set 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.267+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f5048004be0 con 0x7f50580aeca0 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f5048084a80 con 0x7f50580aeca0 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f5054ff9640 10 client.4312.objecter ms_dispatch 0x7f5058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f5054ff9640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f5054ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f5054ff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.268+0000 7f50577fe640 10 monclient: get_auth_request con 0x7f504406df10 auth_method 0 2024-03-29T11:35:03.267 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.269+0000 7f5054ff9640 10 client.4312.objecter ms_handle_connect 0x7f504406df10 2024-03-29T11:35:03.272 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.276+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5048050360 con 0x7f50580aeca0 2024-03-29T11:35:03.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.276+0000 7f5054ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.273 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.276+0000 7f5054ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:03.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.604+0000 7f505e2b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.604+0000 7f505e2b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.604+0000 7f505e2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:03.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.604+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f505813c4a0 con 0x7f50580aeca0 2024-03-29T11:35:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.605+0000 7f5054ff9640 1 -- 172.21.15.45:0/3661117825 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f5048052290 con 0x7f50580aeca0 2024-03-29T11:35:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.605+0000 7f5054ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.603 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.605+0000 7f5054ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:03.606 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:03.606 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 10 librados: watch_flush enter 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 10 librados: watch_flush exit 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 >> v1:172.21.15.190:6816/1411598462 conn(0x7f504406df10 legacy=0x7f50440703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 10 monclient: shutdown 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 >> v1:172.21.15.190:6789/0 conn(0x7f50580aeca0 legacy=0x7f50581cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 shutdown_connections 2024-03-29T11:35:03.610 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.612+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 >> 172.21.15.45:0/3661117825 conn(0x7f505806b660 msgr2=0x7f50580a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:03.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.614+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 shutdown_connections 2024-03-29T11:35:03.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.614+0000 7f505e2b5640 1 -- 172.21.15.45:0/3661117825 wait complete. 2024-03-29T11:35:03.611 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:03.614+0000 7f505e2b5640 1 librados: shutdown 2024-03-29T11:35:03.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:10.437 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:10.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.588+0000 7fe75f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:10.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.588+0000 7fe75f59e640 10 monclient: build_initial_monmap 2024-03-29T11:35:10.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.588+0000 7fe75f59e640 10 monclient: monmap: 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:10.589583+0000 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:10.589583+0000 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:10.587 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.591+0000 7fe75f59e640 1 Processor -- start 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.591+0000 7fe75f59e640 1 -- start start 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.591+0000 7fe75f59e640 10 monclient: init 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:10.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7600a9750 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7600aad20 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe7600a9b30 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600af4d0 con 0x7fe7600aad20 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600af7b0 con 0x7fe7600a9750 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600acab0 con 0x7fe7600a9b30 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:10.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.592+0000 7fe75f59e640 10 monclient(hunting): authenticate will time out at 966.831317s 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75ed9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe7600a9b30 0x7fe760149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41177/0 (socket says 172.21.15.45:41177) 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75ed9d640 1 -- 172.21.15.45:0/529626755 learned_addr learned my addr 172.21.15.45:0/529626755 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585825094 0 0) 0x7fe7600acab0 con 0x7fe7600a9b30 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:35:10.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe748003520 con 0x7fe7600a9b30 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2667423481 0 0) 0x7fe748003520 con 0x7fe7600a9b30 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 >> v1:172.21.15.190:6789/0 conn(0x7fe7600a9750 legacy=0x7fe7600ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.598+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 >> v1:172.21.15.45:6789/0 conn(0x7fe7600aad20 legacy=0x7fe760141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 1 monclient: found mon.noname-b 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600aba30 con 0x7fe7600a9b30 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: _finish_auth 0 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:10.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe758004d90 con 0x7fe7600a9b30 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: dump: 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:10.598 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe758004ff0 con 0x7fe7600a9b30 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75f59e640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 1 -- 172.21.15.45:0/529626755 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe758004d90 con 0x7fe7600a9b30 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75d59a640 10 monclient: dump: 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.599+0000 7fe75f59e640 4 set_mon_vals no callback set 2024-03-29T11:35:10.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.600+0000 7fe75f59e640 10 monclient: shutdown 2024-03-29T11:35:10.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.600+0000 7fe75f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.600+0000 7fe75f59e640 1 -- 172.21.15.45:0/529626755 >> v1:172.21.15.45:6790/0 conn(0x7fe7600a9b30 legacy=0x7fe760149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.603+0000 7fe75f59e640 1 -- 172.21.15.45:0/529626755 shutdown_connections 2024-03-29T11:35:10.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.603+0000 7fe75f59e640 1 -- 172.21.15.45:0/529626755 >> 172.21.15.45:0/529626755 conn(0x7fe7600673e0 msgr2=0x7fe7600b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.602 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.605+0000 7fe75f59e640 1 -- 172.21.15.45:0/529626755 shutdown_connections 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.605+0000 7fe75f59e640 1 -- 172.21.15.45:0/529626755 wait complete. 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.606+0000 7fe75f59e640 10 monclient: build_initial_monmap 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.606+0000 7fe75f59e640 10 monclient: monmap: 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.605 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.606+0000 7fe75f59e640 1 librados: starting msgr at 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.606+0000 7fe75f59e640 1 librados: starting objecter 2024-03-29T11:35:10.606 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.607+0000 7fe75f59e640 1 Processor -- start 2024-03-29T11:35:10.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.607+0000 7fe75f59e640 1 -- start start 2024-03-29T11:35:10.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.607+0000 7fe75f59e640 1 librados: setting wanted keys 2024-03-29T11:35:10.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.607+0000 7fe75f59e640 1 librados: calling monclient init 2024-03-29T11:35:10.607 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.607+0000 7fe75f59e640 10 monclient: init 2024-03-29T11:35:10.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:10.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7600a9750 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe7600a9b30 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7600aad20 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe760071fd0 con 0x7fe7600aad20 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7601e75a0 con 0x7fe7600a9b30 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7601e7030 con 0x7fe7600a9750 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:10.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.608+0000 7fe75f59e640 10 monclient(hunting): authenticate will time out at 966.847904s 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.613+0000 7fe75ed9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe7600aad20 0x7fe7601ef370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59627/0 (socket says 172.21.15.45:59627) 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.613+0000 7fe75ed9d640 1 -- 172.21.15.45:0/2264386283 learned_addr learned my addr 172.21.15.45:0/2264386283 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 client.?.objecter ms_handle_connect 0x7fe7600a9b30 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 client.?.objecter ms_handle_connect 0x7fe7600aad20 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.612 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352857583 0 0) 0x7fe7601e75a0 con 0x7fe7600a9b30 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740004eb0 con 0x7fe7600a9b30 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127442288 0 0) 0x7fe760071fd0 con 0x7fe7600aad20 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:35:10.613 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.614+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7601e75a0 con 0x7fe7600aad20 2024-03-29T11:35:10.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 client.?.objecter ms_handle_connect 0x7fe7600a9750 2024-03-29T11:35:10.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255545017 0 0) 0x7fe7601e7030 con 0x7fe7600a9750 2024-03-29T11:35:10.614 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760071fd0 con 0x7fe7600a9750 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.616+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879033601 0 0) 0x7fe760071fd0 con 0x7fe7600a9750 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7601e7030 con 0x7fe7600a9750 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe754001510 con 0x7fe7600a9750 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient(hunting): dump: 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.615 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.616 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:10.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 640206940 0 0) 0x7fe7601e7030 con 0x7fe7600a9750 2024-03-29T11:35:10.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 >> v1:172.21.15.190:6789/0 conn(0x7fe7600a9b30 legacy=0x7fe7601d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 >> v1:172.21.15.45:6789/0 conn(0x7fe7600aad20 legacy=0x7fe7601ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.617 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 monclient: found mon.c 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7601e71c0 con 0x7fe7600a9750 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe74effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 10 monclient: _renew_subs 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76013bb10 con 0x7fe7600a9750 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 10 monclient: _renew_subs 2024-03-29T11:35:10.618 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76013f5e0 con 0x7fe7600a9750 2024-03-29T11:35:10.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.617+0000 7fe75f59e640 1 librados: init done 2024-03-29T11:35:10.619 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.620+0000 7fe75f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.620+0000 7fe75f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.620+0000 7fe75f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.620+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7601e7750 con 0x7fe7600a9750 2024-03-29T11:35:10.620 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe754001a00 con 0x7fe7600a9750 2024-03-29T11:35:10.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:10.621 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe754003890 con 0x7fe7600a9750 2024-03-29T11:35:10.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:10.622 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.621+0000 7fe74effd640 10 monclient: dump: 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.622+0000 7fe75cd99640 4 set_mon_vals no callback set 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.622+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe754019860 con 0x7fe7600a9750 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.623+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe754083720 con 0x7fe7600a9750 2024-03-29T11:35:10.623 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.623+0000 7fe74effd640 10 client.4349.objecter ms_dispatch 0x7fe760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:10.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.623+0000 7fe74effd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:10.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.623+0000 7fe74effd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:10.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.623+0000 7fe74effd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:35:10.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.625+0000 7fe75dd9b640 10 monclient: get_auth_request con 0x7fe74006de90 auth_method 0 2024-03-29T11:35:10.624 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.626+0000 7fe74effd640 10 client.4349.objecter ms_handle_connect 0x7fe74006de90 2024-03-29T11:35:10.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.636+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7601e7750 con 0x7fe7600a9750 2024-03-29T11:35:10.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.636+0000 7fe74effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.634 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.636+0000 7fe74effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:10.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.977+0000 7fe75f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.977+0000 7fe75f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.977+0000 7fe75f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:10.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.977+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7601e7750 con 0x7fe7600a9750 2024-03-29T11:35:10.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.978+0000 7fe74effd640 1 -- 172.21.15.45:0/2264386283 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe7601e7750 con 0x7fe7600a9750 2024-03-29T11:35:10.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.978+0000 7fe74effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.978+0000 7fe74effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:10.976 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:10.977 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:10.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.981+0000 7fe75f59e640 10 librados: watch_flush enter 2024-03-29T11:35:10.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.981+0000 7fe75f59e640 10 librados: watch_flush exit 2024-03-29T11:35:10.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.981+0000 7fe75f59e640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:35:10.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.981+0000 7fe75f59e640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:35:10.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe74006de90 legacy=0x7fe740070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 10 monclient: shutdown 2024-03-29T11:35:10.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.982 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 >> v1:172.21.15.45:6790/0 conn(0x7fe7600a9750 legacy=0x7fe7601e3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 shutdown_connections 2024-03-29T11:35:10.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.984+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 >> 172.21.15.45:0/2264386283 conn(0x7fe7600673e0 msgr2=0x7fe7600b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.985+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 shutdown_connections 2024-03-29T11:35:10.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.985+0000 7fe75f59e640 1 -- 172.21.15.45:0/2264386283 wait complete. 2024-03-29T11:35:10.985 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:10.987+0000 7fe75f59e640 1 librados: shutdown 2024-03-29T11:35:11.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:17.811 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:17.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.961+0000 7f9281272640 10 monclient: get_monmap_and_config 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.961+0000 7f9281272640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.961+0000 7f9281272640 10 monclient: monmap: 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:17.962626+0000 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:17.962626+0000 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:17.960 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.962+0000 7f9281272640 1 Processor -- start 2024-03-29T11:35:17.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.962+0000 7f9281272640 1 -- start start 2024-03-29T11:35:17.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.962+0000 7f9281272640 10 monclient: init 2024-03-29T11:35:17.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient(hunting): picked mon.noname-c con 0x7f9274006a70 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient(hunting): picked mon.noname-b con 0x7f927402b100 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient(hunting): picked mon.noname-a con 0x7f9274029c30 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927402a270 con 0x7f9274029c30 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927402c440 con 0x7f9274006a70 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92740c9440 con 0x7f927402b100 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.963+0000 7f9281272640 10 monclient(hunting): authenticate will time out at 974.202827s 2024-03-29T11:35:17.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.969+0000 7f927bfff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f9274006a70 0x7f92740c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45165/0 (socket says 172.21.15.45:45165) 2024-03-29T11:35:17.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.969+0000 7f927bfff640 1 -- 172.21.15.45:0/98316477 learned_addr learned my addr 172.21.15.45:0/98316477 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.970+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74239397 0 0) 0x7f927402c440 con 0x7f9274006a70 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.970+0000 7f927affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:35:17.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c003520 con 0x7f9274006a70 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308701533 0 0) 0x7f927402a270 con 0x7f9274029c30 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f927402c440 con 0x7f9274029c30 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276996942 0 0) 0x7f926c003520 con 0x7f9274006a70 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 >> v1:172.21.15.45:6790/0 conn(0x7f927402b100 legacy=0x7f92740cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 >> v1:172.21.15.45:6789/0 conn(0x7f9274029c30 legacy=0x7f92740d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.971+0000 7f927affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 1 monclient: found mon.noname-c 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740c95d0 con 0x7f9274006a70 2024-03-29T11:35:17.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f927c04f730 con 0x7f9274006a70 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: dump: 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:17.974 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f9281272640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f9281272640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f927c075f10 con 0x7f9274006a70 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.972+0000 7f927affd640 1 -- 172.21.15.45:0/98316477 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f927c0758b0 con 0x7f9274006a70 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f9281272640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f9281272640 4 set_mon_vals no callback set 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f927affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f927affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f927affd640 10 monclient: dump: 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.975 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f9281272640 10 monclient: shutdown 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f9281272640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.973+0000 7f9281272640 1 -- 172.21.15.45:0/98316477 >> v1:172.21.15.190:6789/0 conn(0x7f9274006a70 legacy=0x7f92740c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.976+0000 7f9281272640 1 -- 172.21.15.45:0/98316477 shutdown_connections 2024-03-29T11:35:17.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.976+0000 7f9281272640 1 -- 172.21.15.45:0/98316477 >> 172.21.15.45:0/98316477 conn(0x7f92740058f0 msgr2=0x7f9274005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.976+0000 7f9281272640 1 -- 172.21.15.45:0/98316477 shutdown_connections 2024-03-29T11:35:17.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.976+0000 7f9281272640 1 -- 172.21.15.45:0/98316477 wait complete. 2024-03-29T11:35:17.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.977+0000 7f9281272640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.977+0000 7f9281272640 10 monclient: monmap: 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.977+0000 7f9281272640 1 librados: starting msgr at 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.977+0000 7f9281272640 1 librados: starting objecter 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.977+0000 7f9281272640 1 Processor -- start 2024-03-29T11:35:17.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.978+0000 7f9281272640 1 -- start start 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.978+0000 7f9281272640 1 librados: setting wanted keys 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.978+0000 7f9281272640 1 librados: calling monclient init 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.978+0000 7f9281272640 10 monclient: init 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient(hunting): picked mon.noname-c con 0x7f9274006a70 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient(hunting): picked mon.noname-b con 0x7f9274029c30 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient(hunting): picked mon.noname-a con 0x7f927402c8f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927416cbe0 con 0x7f927402c8f0 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92740cccb0 con 0x7f9274029c30 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92740cce40 con 0x7f9274006a70 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.979+0000 7f9281272640 10 monclient(hunting): authenticate will time out at 974.218315s 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f927bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9274006a70 0x7f927402d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58023/0 (socket says 172.21.15.45:58023) 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f927bfff640 1 -- 172.21.15.45:0/3799875536 learned_addr learned my addr 172.21.15.45:0/3799875536 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 client.?.objecter ms_handle_connect 0x7f9274006a70 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493471317 0 0) 0x7f92740cce40 con 0x7f9274006a70 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:35:17.987 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.985+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9260004eb0 con 0x7f9274006a70 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370336094 0 0) 0x7f9260004eb0 con 0x7f9274006a70 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92740cce40 con 0x7f9274006a70 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f927c04f400 con 0x7f9274006a70 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:17.988 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 client.?.objecter ms_handle_connect 0x7f927402c8f0 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574963036 0 0) 0x7f92740cce40 con 0x7f9274006a70 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 >> v1:172.21.15.190:6789/0 conn(0x7f9274029c30 legacy=0x7f927416d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 >> v1:172.21.15.45:6789/0 conn(0x7f927402c8f0 legacy=0x7f9274166690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 monclient: found mon.c 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927402dda0 con 0x7f9274006a70 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.986+0000 7f9278ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 10 monclient: _renew_subs 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927416d780 con 0x7f9274006a70 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 10 monclient: _renew_subs 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9274166f80 con 0x7f9274006a70 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.987+0000 7f9281272640 1 librados: init done 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.988+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f927c04fdd0 con 0x7f9274006a70 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.988+0000 7f9278ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.988+0000 7f927a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:17.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.990+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f927c067bf0 con 0x7f9274006a70 2024-03-29T11:35:17.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.990+0000 7f9278ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.990+0000 7f9278ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.990+0000 7f9278ff9640 10 monclient: dump: 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.990+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f927c067e10 con 0x7f9274006a70 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f927b7fe640 10 monclient: get_auth_request con 0x7f926006db20 auth_method 0 2024-03-29T11:35:17.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f927c0ff880 con 0x7f9274006a70 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 10 client.4352.objecter ms_dispatch 0x7f927c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9278ff9640 10 client.4352.objecter ms_handle_connect 0x7f926006db20 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9281272640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9281272640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9281272640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:17.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:17.992+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c005080 con 0x7f9274006a70 2024-03-29T11:35:18.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.002+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f927c0cb1e0 con 0x7f9274006a70 2024-03-29T11:35:18.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.002+0000 7f9278ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:18.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.002+0000 7f9278ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:18.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.330+0000 7f9281272640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.330+0000 7f9281272640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.330+0000 7f9281272640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:18.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.330+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f925c005330 con 0x7f9274006a70 2024-03-29T11:35:18.332 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:18.332 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:18.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.334+0000 7f9278ff9640 1 -- 172.21.15.45:0/3799875536 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f927c0cd110 con 0x7f9274006a70 2024-03-29T11:35:18.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.334+0000 7f9278ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.334+0000 7f9278ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:18.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.337+0000 7f9281272640 10 librados: watch_flush enter 2024-03-29T11:35:18.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 10 librados: watch_flush exit 2024-03-29T11:35:18.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:35:18.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:35:18.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 >> v1:172.21.15.190:6816/1411598462 conn(0x7f926006db20 legacy=0x7f926006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 10 monclient: shutdown 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 >> v1:172.21.15.45:6790/0 conn(0x7f9274006a70 legacy=0x7f927402d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 shutdown_connections 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 >> 172.21.15.45:0/3799875536 conn(0x7f92740058f0 msgr2=0x7f92740c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 shutdown_connections 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 -- 172.21.15.45:0/3799875536 wait complete. 2024-03-29T11:35:18.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:18.338+0000 7f9281272640 1 librados: shutdown 2024-03-29T11:35:18.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:25.162 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.311+0000 7f22776c0640 10 monclient: get_monmap_and_config 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.311+0000 7f22776c0640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.311+0000 7f22776c0640 10 monclient: monmap: 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:25.313109+0000 2024-03-29T11:35:25.310 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:25.313109+0000 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.312+0000 7f22776c0640 1 Processor -- start 2024-03-29T11:35:25.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.313+0000 7f22776c0640 1 -- start start 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.313+0000 7f22776c0640 10 monclient: init 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-a con 0x7f22701457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-c con 0x7f227013f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-b con 0x7f22700a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22700aaef0 con 0x7f22701457f0 2024-03-29T11:35:25.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2270145bd0 con 0x7f227013f650 2024-03-29T11:35:25.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2270145eb0 con 0x7f22700a9d70 2024-03-29T11:35:25.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f22776c0640 10 monclient(hunting): authenticate will time out at 981.553419s 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f2274c5b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f227013f650 0x7f22700b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:35736/0 (socket says 172.21.15.45:35736) 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.314+0000 7f2274c5b640 1 -- 172.21.15.45:0/2573280469 learned_addr learned my addr 172.21.15.45:0/2573280469 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688711578 0 0) 0x7f22700aaef0 con 0x7f22701457f0 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2258003520 con 0x7f22701457f0 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110614959 0 0) 0x7f2270145bd0 con 0x7f227013f650 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22700aaef0 con 0x7f227013f650 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323777523 0 0) 0x7f2270145eb0 con 0x7f22700a9d70 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:35:25.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.315+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270145bd0 con 0x7f22700a9d70 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015263154 0 0) 0x7f22700aaef0 con 0x7f227013f650 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 >> v1:172.21.15.45:6790/0 conn(0x7f22700a9d70 legacy=0x7f227014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 >> v1:172.21.15.45:6789/0 conn(0x7f22701457f0 legacy=0x7f227013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 monclient: found mon.noname-c 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2270146340 con 0x7f227013f650 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2264001670 con 0x7f227013f650 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 10 monclient: dump: 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f22776c0640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f22776c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:25.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.316+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22640017d0 con 0x7f227013f650 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f226ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f226ffff640 1 -- 172.21.15.45:0/2573280469 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2264001be0 con 0x7f227013f650 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f226ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f226ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f226ffff640 10 monclient: dump: 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.319 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.320 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f22776c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:25.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f22776c0640 4 set_mon_vals no callback set 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f22776c0640 10 monclient: shutdown 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f22776c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.317+0000 7f22776c0640 1 -- 172.21.15.45:0/2573280469 >> v1:172.21.15.190:6789/0 conn(0x7f227013f650 legacy=0x7f22700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.318+0000 7f22776c0640 1 -- 172.21.15.45:0/2573280469 shutdown_connections 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.318+0000 7f22776c0640 1 -- 172.21.15.45:0/2573280469 >> 172.21.15.45:0/2573280469 conn(0x7f2270066a00 msgr2=0x7f2270066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.318+0000 7f22776c0640 1 -- 172.21.15.45:0/2573280469 shutdown_connections 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.318+0000 7f22776c0640 1 -- 172.21.15.45:0/2573280469 wait complete. 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.319+0000 7f22776c0640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.319+0000 7f22776c0640 10 monclient: monmap: 2024-03-29T11:35:25.328 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.319+0000 7f22776c0640 1 librados: starting msgr at 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 1 librados: starting objecter 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 1 Processor -- start 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 1 -- start start 2024-03-29T11:35:25.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 1 librados: setting wanted keys 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 1 librados: calling monclient init 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.320+0000 7f22776c0640 10 monclient: init 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-a con 0x7f22700a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-c con 0x7f227013f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient(hunting): picked mon.noname-b con 0x7f22701457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22701d72d0 con 0x7f22700a9d70 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22701387a0 con 0x7f22701457f0 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22700ad1e0 con 0x7f227013f650 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f22776c0640 10 monclient(hunting): authenticate will time out at 981.560455s 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f2275c5d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f22701457f0 0x7f2270138090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:36664/0 (socket says 172.21.15.45:36664) 2024-03-29T11:35:25.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f2275c5d640 1 -- 172.21.15.45:0/3838081696 learned_addr learned my addr 172.21.15.45:0/3838081696 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f226dffb640 10 client.?.objecter ms_handle_connect 0x7f22701457f0 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.321+0000 7f226dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 client.?.objecter ms_handle_connect 0x7f22700a9d70 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 client.?.objecter ms_handle_connect 0x7f227013f650 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151764455 0 0) 0x7f22701387a0 con 0x7f22701457f0 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c004eb0 con 0x7f22701457f0 2024-03-29T11:35:25.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075113109 0 0) 0x7f22700ad1e0 con 0x7f227013f650 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22701387a0 con 0x7f227013f650 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025897621 0 0) 0x7f22701d72d0 con 0x7f22700a9d70 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22700ad1e0 con 0x7f22700a9d70 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338243097 0 0) 0x7f225c004eb0 con 0x7f22701457f0 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22701d72d0 con 0x7f22701457f0 2024-03-29T11:35:25.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2260001c80 con 0x7f22701457f0 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.322+0000 7f226dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): dump: 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268146199 0 0) 0x7f22701387a0 con 0x7f227013f650 2024-03-29T11:35:25.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f225c004eb0 con 0x7f227013f650 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2264001ae0 con 0x7f227013f650 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): dump: 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.334 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665439787 0 0) 0x7f22700ad1e0 con 0x7f22700a9d70 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22701387a0 con 0x7f22700a9d70 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2268001850 con 0x7f22700a9d70 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient(hunting): dump: 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.335 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1659533243 0 0) 0x7f22701d72d0 con 0x7f22701457f0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 >> v1:172.21.15.45:6790/0 conn(0x7f227013f650 legacy=0x7f2270137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 >> v1:172.21.15.45:6789/0 conn(0x7f22700a9d70 legacy=0x7f22701d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 monclient: found mon.b 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22700ad4c0 con 0x7f22701457f0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f226dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 10 monclient: _renew_subs 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227013fda0 con 0x7f22701457f0 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 10 monclient: _renew_subs 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22701d8050 con 0x7f22701457f0 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.323+0000 7f22776c0640 1 librados: init done 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2260003990 con 0x7f22701457f0 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2260003d60 con 0x7f22701457f0 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226f7fe640 4 set_mon_vals no callback set 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.324+0000 7f226dffb640 10 monclient: dump: 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:25.337 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.326+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f2260003ec0 con 0x7f22701457f0 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.326+0000 7f22776c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.326+0000 7f22776c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.326+0000 7f22776c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.326+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2250005080 con 0x7f22701457f0 2024-03-29T11:35:25.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2260084db0 con 0x7f22701457f0 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 10 client.4324.objecter ms_dispatch 0x7f2270000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f227545c640 10 monclient: get_auth_request con 0x7f225c06e120 auth_method 0 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.327+0000 7f226dffb640 10 client.4324.objecter ms_handle_connect 0x7f225c06e120 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.334+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2260050690 con 0x7f22701457f0 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.334+0000 7f226dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.334+0000 7f226dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:25.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.712+0000 7f22776c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.712+0000 7f22776c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.712+0000 7f22776c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:25.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.712+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2250005330 con 0x7f22701457f0 2024-03-29T11:35:25.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.713+0000 7f226dffb640 1 -- 172.21.15.45:0/3838081696 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f22600525c0 con 0x7f22701457f0 2024-03-29T11:35:25.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.713+0000 7f226dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.713+0000 7f226dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:25.712 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:25.712 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 10 librados: watch_flush enter 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 10 librados: watch_flush exit 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 >> v1:172.21.15.190:6816/1411598462 conn(0x7f225c06e120 legacy=0x7f225c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 10 monclient: shutdown 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.717+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 >> v1:172.21.15.190:6789/0 conn(0x7f22701457f0 legacy=0x7f2270138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.718+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 shutdown_connections 2024-03-29T11:35:25.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.718+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 >> 172.21.15.45:0/3838081696 conn(0x7f2270066a00 msgr2=0x7f22700b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.718+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 shutdown_connections 2024-03-29T11:35:25.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.718+0000 7f22776c0640 1 -- 172.21.15.45:0/3838081696 wait complete. 2024-03-29T11:35:25.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:25.718+0000 7f22776c0640 1 librados: shutdown 2024-03-29T11:35:25.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:32.537 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:32.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.706+0000 7f72f3354640 10 monclient: get_monmap_and_config 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.706+0000 7f72f3354640 10 monclient: build_initial_monmap 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.706+0000 7f72f3354640 10 monclient: monmap: 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:32.707477+0000 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:32.707477+0000 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:32.705 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.707+0000 7f72f3354640 1 Processor -- start 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.707+0000 7f72f3354640 1 -- start start 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.707+0000 7f72f3354640 10 monclient: init 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-c con 0x7f72ec0b3ab0 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:32.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-a con 0x7f72ec0b3e90 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-b con 0x7f72ec143cf0 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec06e0a0 con 0x7f72ec0b3e90 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0b4500 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec071830 con 0x7f72ec143cf0 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:32.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.708+0000 7f72f3354640 10 monclient(hunting): authenticate will time out at 988.947843s 2024-03-29T11:35:32.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.711+0000 7f72f2b53640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f72ec143cf0 0x7f72ec151bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51196/0 (socket says 172.21.15.45:51196) 2024-03-29T11:35:32.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.711+0000 7f72f2b53640 1 -- 172.21.15.45:0/2441512190 learned_addr learned my addr 172.21.15.45:0/2441512190 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:32.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692423018 0 0) 0x7f72ec0b4500 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d8003520 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189655000 0 0) 0x7f72ec071830 con 0x7f72ec143cf0 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:35:32.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.715+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec0b4500 con 0x7f72ec143cf0 2024-03-29T11:35:32.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39757088 0 0) 0x7f72d8003520 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 >> v1:172.21.15.45:6790/0 conn(0x7f72ec143cf0 legacy=0x7f72ec151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.715 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 >> v1:172.21.15.45:6789/0 conn(0x7f72ec0b3e90 legacy=0x7f72ec1435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: _finish_hunting 0 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 monclient: found mon.noname-c 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0719c0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: _finish_auth 0 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: _check_auth_tickets 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f72e8001690 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: dump: 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.716 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f3354640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f3354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72e80017f0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 1 -- 172.21.15.45:0/2441512190 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f72e8001c00 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f3354640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f3354640 4 set_mon_vals no callback set 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.716+0000 7f72f1350640 10 monclient: dump: 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:32.718 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.717+0000 7f72f3354640 10 monclient: shutdown 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.717+0000 7f72f3354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.717+0000 7f72f3354640 1 -- 172.21.15.45:0/2441512190 >> v1:172.21.15.190:6789/0 conn(0x7f72ec0b3ab0 legacy=0x7f72ec146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.718+0000 7f72f3354640 1 -- 172.21.15.45:0/2441512190 shutdown_connections 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.718+0000 7f72f3354640 1 -- 172.21.15.45:0/2441512190 >> 172.21.15.45:0/2441512190 conn(0x7f72ec06d140 msgr2=0x7f72ec066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.718+0000 7f72f3354640 1 -- 172.21.15.45:0/2441512190 shutdown_connections 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.718+0000 7f72f3354640 1 -- 172.21.15.45:0/2441512190 wait complete. 2024-03-29T11:35:32.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.719+0000 7f72f3354640 10 monclient: build_initial_monmap 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.719+0000 7f72f3354640 10 monclient: monmap: 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.719+0000 7f72f3354640 1 librados: starting msgr at 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 1 librados: starting objecter 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 1 Processor -- start 2024-03-29T11:35:32.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 1 -- start start 2024-03-29T11:35:32.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 1 librados: setting wanted keys 2024-03-29T11:35:32.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 1 librados: calling monclient init 2024-03-29T11:35:32.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.720+0000 7f72f3354640 10 monclient: init 2024-03-29T11:35:32.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.721+0000 7f72f3354640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:32.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.721+0000 7f72f3354640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:32.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.721+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-a con 0x7f72ec0b3ab0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-b con 0x7f72ec0b3e90 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 10 monclient(hunting): picked mon.noname-c con 0x7f72ec143cf0 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec1e3f90 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec1e4b80 con 0x7f72ec0b3e90 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec1e4d10 con 0x7f72ec143cf0 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.722+0000 7f72f3354640 10 monclient(hunting): authenticate will time out at 988.961441s 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.724+0000 7f72f2352640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f72ec0b3ab0 0x7f72ec1e4470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:33589/0 (socket says 172.21.15.45:33589) 2024-03-29T11:35:32.723 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.725+0000 7f72f2352640 1 -- 172.21.15.45:0/3240339649 learned_addr learned my addr 172.21.15.45:0/3240339649 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:32.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.725+0000 7f72e2ffd640 10 client.?.objecter ms_handle_connect 0x7f72ec0b3ab0 2024-03-29T11:35:32.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.725+0000 7f72e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.725+0000 7f72e2ffd640 10 client.?.objecter ms_handle_connect 0x7f72ec143cf0 2024-03-29T11:35:32.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.725+0000 7f72e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565818383 0 0) 0x7f72ec1e4d10 con 0x7f72ec143cf0 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d0004eb0 con 0x7f72ec143cf0 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 client.?.objecter ms_handle_connect 0x7f72ec0b3e90 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838939276 0 0) 0x7f72ec1e3f90 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.729+0000 7f72e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec1e4d10 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731491012 0 0) 0x7f72ec1e4b80 con 0x7f72ec0b3e90 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec1e3f90 con 0x7f72ec0b3e90 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700363688 0 0) 0x7f72d0004eb0 con 0x7f72ec143cf0 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ec1e4b80 con 0x7f72ec143cf0 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f72e4001a70 con 0x7f72ec143cf0 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.730+0000 7f72e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665849320 0 0) 0x7f72ec1e4d10 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d0004eb0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f72e8001b40 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2133073440 0 0) 0x7f72d0004eb0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 >> v1:172.21.15.45:6790/0 conn(0x7f72ec143cf0 legacy=0x7f72ec1ef650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 >> v1:172.21.15.190:6789/0 conn(0x7f72ec0b3e90 legacy=0x7f72ec1e1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 monclient: found mon.a 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec144bb0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.731+0000 7f72e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72e80038a0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f72e8003cb0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.732+0000 7f72e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72e2ffd640 10 monclient: dump: 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:32.736 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f0b4f640 4 set_mon_vals no callback set 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 10 monclient: _renew_subs 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec144e00 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 10 monclient: _renew_subs 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec1e1900 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.733+0000 7f72f3354640 1 librados: init done 2024-03-29T11:35:32.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.736+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f72e8003e10 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f72e8084ae0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72e2ffd640 10 client.4356.objecter ms_dispatch 0x7f72ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72e2ffd640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72e2ffd640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72e2ffd640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72f3354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72f3354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72f3354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.737+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec1e3df0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.739+0000 7f72f1b51640 10 monclient: get_auth_request con 0x7f72d006dfb0 auth_method 0 2024-03-29T11:35:32.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.740+0000 7f72e2ffd640 10 client.4356.objecter ms_handle_connect 0x7f72d006dfb0 2024-03-29T11:35:32.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.750+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72ec1e3df0 con 0x7f72ec0b3ab0 2024-03-29T11:35:32.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.750+0000 7f72e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.748 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:32.750+0000 7f72e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:33.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.121+0000 7f72f3354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.121+0000 7f72f3354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.121+0000 7f72f3354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:35:33.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.121+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72ec1e3df0 con 0x7f72ec0b3ab0 2024-03-29T11:35:33.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.123+0000 7f72e2ffd640 1 -- 172.21.15.45:0/3240339649 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f72ec1e3df0 con 0x7f72ec0b3ab0 2024-03-29T11:35:33.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.123+0000 7f72e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:33.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.123+0000 7f72e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:33.122 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:33.122 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 10 librados: watch_flush enter 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 10 librados: watch_flush exit 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 >> v1:172.21.15.190:6816/1411598462 conn(0x7f72d006dfb0 legacy=0x7f72d0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 10 monclient: shutdown 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 >> v1:172.21.15.45:6789/0 conn(0x7f72ec0b3ab0 legacy=0x7f72ec1e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 shutdown_connections 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.126+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 >> 172.21.15.45:0/3240339649 conn(0x7f72ec06d140 msgr2=0x7f72ec066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.127+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 shutdown_connections 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.127+0000 7f72f3354640 1 -- 172.21.15.45:0/3240339649 wait complete. 2024-03-29T11:35:33.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:33.127+0000 7f72f3354640 1 librados: shutdown 2024-03-29T11:35:33.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:39.946 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:40.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.095+0000 7f0533d15640 10 monclient: get_monmap_and_config 2024-03-29T11:35:40.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.095+0000 7f0533d15640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.093 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.095+0000 7f0533d15640 10 monclient: monmap: 2024-03-29T11:35:40.093 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:40.096302+0000 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:40.096302+0000 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:40.094 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.096+0000 7f0533d15640 1 Processor -- start 2024-03-29T11:35:40.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.096+0000 7f0533d15640 1 -- start start 2024-03-29T11:35:40.095 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.096+0000 7f0533d15640 10 monclient: init 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-a con 0x7f052c1457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-b con 0x7f052c13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-c con 0x7f052c0a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0aaef0 con 0x7f052c1457f0 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c145bd0 con 0x7f052c0a9d70 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c145eb0 con 0x7f052c13f650 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f0533d15640 10 monclient(hunting): authenticate will time out at 996.336586s 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f05312b0640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f052c13f650 0x7f052c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53882/0 (socket says 172.21.15.45:53882) 2024-03-29T11:35:40.096 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.097+0000 7f05312b0640 1 -- 172.21.15.45:0/1414683106 learned_addr learned my addr 172.21.15.45:0/1414683106 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:40.098 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919839394 0 0) 0x7f052c145bd0 con 0x7f052c0a9d70 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0518003520 con 0x7f052c0a9d70 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887129462 0 0) 0x7f052c145eb0 con 0x7f052c13f650 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c145bd0 con 0x7f052c13f650 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567110884 0 0) 0x7f052c145bd0 con 0x7f052c13f650 2024-03-29T11:35:40.099 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 >> v1:172.21.15.190:6789/0 conn(0x7f052c0a9d70 legacy=0x7f052c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 >> v1:172.21.15.45:6789/0 conn(0x7f052c1457f0 legacy=0x7f052c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 1 monclient: found mon.noname-b 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.098+0000 7f0530aaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c146340 con 0x7f052c13f650 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0520001670 con 0x7f052c13f650 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: dump: 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:40.100 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0533d15640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0533d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0520001890 con 0x7f052c13f650 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 1 -- 172.21.15.45:0/1414683106 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0520001c20 con 0x7f052c13f650 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0533d15640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:40.101 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0533d15640 4 set_mon_vals no callback set 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.099+0000 7f0530aaf640 10 monclient: dump: 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 10 monclient: shutdown 2024-03-29T11:35:40.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 1 -- 172.21.15.45:0/1414683106 >> v1:172.21.15.45:6790/0 conn(0x7f052c13f650 legacy=0x7f052c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 1 -- 172.21.15.45:0/1414683106 shutdown_connections 2024-03-29T11:35:40.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 1 -- 172.21.15.45:0/1414683106 >> 172.21.15.45:0/1414683106 conn(0x7f052c066a00 msgr2=0x7f052c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 1 -- 172.21.15.45:0/1414683106 shutdown_connections 2024-03-29T11:35:40.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.100+0000 7f0533d15640 1 -- 172.21.15.45:0/1414683106 wait complete. 2024-03-29T11:35:40.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.101+0000 7f0533d15640 10 monclient: build_initial_monmap 2024-03-29T11:35:40.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.101+0000 7f0533d15640 10 monclient: monmap: 2024-03-29T11:35:40.107 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:40.107 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.101+0000 7f0533d15640 1 librados: starting msgr at 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.101+0000 7f0533d15640 1 librados: starting objecter 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.101+0000 7f0533d15640 1 Processor -- start 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 1 -- start start 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 1 librados: setting wanted keys 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 1 librados: calling monclient init 2024-03-29T11:35:40.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 10 monclient: init 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.102+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-c con 0x7f052c0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-a con 0x7f052c13f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 10 monclient(hunting): picked mon.noname-b con 0x7f052c1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c142960 con 0x7f052c13f650 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c142c40 con 0x7f052c1457f0 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c141240 con 0x7f052c0a9d70 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0533d15640 10 monclient(hunting): authenticate will time out at 996.342213s 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0531ab1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f052c0a9d70 0x7f052c138d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54372/0 (socket says 172.21.15.45:54372) 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f0531ab1640 1 -- 172.21.15.45:0/671201277 learned_addr learned my addr 172.21.15.45:0/671201277 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter ms_handle_connect 0x7f052c0a9d70 2024-03-29T11:35:40.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter ms_handle_connect 0x7f052c1457f0 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter ms_handle_connect 0x7f052c13f650 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995163891 0 0) 0x7f052c141240 con 0x7f052c0a9d70 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510004eb0 con 0x7f052c0a9d70 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806360086 0 0) 0x7f052c142960 con 0x7f052c13f650 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.103+0000 7f051e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c141240 con 0x7f052c13f650 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303001787 0 0) 0x7f052c142c40 con 0x7f052c1457f0 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c142960 con 0x7f052c1457f0 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716452127 0 0) 0x7f0510004eb0 con 0x7f052c0a9d70 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f052c142c40 con 0x7f052c0a9d70 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0524001cb0 con 0x7f052c0a9d70 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:40.111 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785839040 0 0) 0x7f052c142960 con 0x7f052c1457f0 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0510004eb0 con 0x7f052c1457f0 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0528001880 con 0x7f052c1457f0 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3570859434 0 0) 0x7f052c142c40 con 0x7f052c0a9d70 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 >> v1:172.21.15.190:6789/0 conn(0x7f052c1457f0 legacy=0x7f052c140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.104+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 >> v1:172.21.15.45:6789/0 conn(0x7f052c13f650 legacy=0x7f052c141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 1 monclient: found mon.c 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c1413d0 con 0x7f052c0a9d70 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 10 monclient: _renew_subs 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c144990 con 0x7f052c0a9d70 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:40.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 10 monclient: _renew_subs 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c132e60 con 0x7f052c0a9d70 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f0533d15640 1 librados: init done 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0524001e10 con 0x7f052c0a9d70 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0524003a50 con 0x7f052c0a9d70 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051ffff640 4 set_mon_vals no callback set 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.105+0000 7f051e7fc640 10 monclient: dump: 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:40.115 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.107+0000 7f0533d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.107+0000 7f0533d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.107+0000 7f0533d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.107+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514005080 con 0x7f052c0a9d70 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.108+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f0524003bb0 con 0x7f052c0a9d70 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.108+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f0524084c50 con 0x7f052c0a9d70 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.108+0000 7f051e7fc640 10 client.4370.objecter ms_dispatch 0x7f052c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.108+0000 7f051e7fc640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.108+0000 7f051e7fc640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:40.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.109+0000 7f051e7fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:35:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.109+0000 7f05312b0640 10 monclient: get_auth_request con 0x7f051006df90 auth_method 0 2024-03-29T11:35:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.109+0000 7f051e7fc640 10 client.4370.objecter ms_handle_connect 0x7f051006df90 2024-03-29T11:35:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.116+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0524050360 con 0x7f052c0a9d70 2024-03-29T11:35:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.116+0000 7f051e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:40.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.116+0000 7f051e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:40.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.499+0000 7f0533d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.499+0000 7f0533d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.499+0000 7f0533d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:40.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.499+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0514005330 con 0x7f052c0a9d70 2024-03-29T11:35:40.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.500+0000 7f051e7fc640 1 -- 172.21.15.45:0/671201277 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f0524052460 con 0x7f052c0a9d70 2024-03-29T11:35:40.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.500+0000 7f051e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.500+0000 7f051e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:40.499 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:40.499 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:40.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 10 librados: watch_flush enter 2024-03-29T11:35:40.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 10 librados: watch_flush exit 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 >> v1:172.21.15.190:6816/1411598462 conn(0x7f051006df90 legacy=0x7f0510070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 10 monclient: shutdown 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 >> v1:172.21.15.45:6790/0 conn(0x7f052c0a9d70 legacy=0x7f052c138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 shutdown_connections 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 >> 172.21.15.45:0/671201277 conn(0x7f052c066a00 msgr2=0x7f052c0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 shutdown_connections 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 -- 172.21.15.45:0/671201277 wait complete. 2024-03-29T11:35:40.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:40.504+0000 7f0533d15640 1 librados: shutdown 2024-03-29T11:35:40.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:47.323 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.457+0000 7f64c7ad9640 10 monclient: get_monmap_and_config 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.457+0000 7f64c7ad9640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.458+0000 7f64c7ad9640 10 monclient: monmap: 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:47.459197+0000 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:47.459197+0000 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.458 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.459+0000 7f64c7ad9640 1 Processor -- start 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.459+0000 7f64c7ad9640 1 -- start start 2024-03-29T11:35:47.459 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.459+0000 7f64c7ad9640 10 monclient: init 2024-03-29T11:35:47.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:47.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00a9930 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:47.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f64c0149dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:47.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f64c014ff40 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00aae60 con 0x7f64c0149dc0 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00b4310 con 0x7f64c014ff40 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00afdd0 con 0x7f64c00a9930 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c7ad9640 10 monclient(hunting): authenticate will time out at 1003.699359s 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c5875640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f64c00a9930 0x7f64c0143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39778/0 (socket says 172.21.15.45:39778) 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c5875640 1 -- 172.21.15.45:0/1417561315 learned_addr learned my addr 172.21.15.45:0/1417561315 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813377298 0 0) 0x7f64c00afdd0 con 0x7f64c00a9930 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:35:47.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac003520 con 0x7f64c00a9930 2024-03-29T11:35:47.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394572039 0 0) 0x7f64ac003520 con 0x7f64c00a9930 2024-03-29T11:35:47.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 >> v1:172.21.15.190:6789/0 conn(0x7f64c014ff40 legacy=0x7f64c0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:47.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.460+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 >> v1:172.21.15.45:6789/0 conn(0x7f64c0149dc0 legacy=0x7f64c014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: _finish_hunting 0 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 1 monclient: found mon.noname-b 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c014a950 con 0x7f64c00a9930 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: _finish_auth 0 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: _check_auth_tickets 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b0001690 con 0x7f64c00a9930 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: dump: 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:47.463 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c7ad9640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c7ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b0001860 con 0x7f64c00a9930 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c4873640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c7ad9640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:47.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.464+0000 7f64c7ad9640 4 set_mon_vals no callback set 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c4873640 1 -- 172.21.15.45:0/1417561315 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b0001cb0 con 0x7f64c00a9930 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c4873640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c4873640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c4873640 10 monclient: dump: 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:47.466 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c7ad9640 10 monclient: shutdown 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c7ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1417561315 >> v1:172.21.15.45:6790/0 conn(0x7f64c00a9930 legacy=0x7f64c0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1417561315 shutdown_connections 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.465+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1417561315 >> 172.21.15.45:0/1417561315 conn(0x7f64c00b35e0 msgr2=0x7f64c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.466+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1417561315 shutdown_connections 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.466+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1417561315 wait complete. 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 10 monclient: monmap: 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.467 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 1 librados: starting msgr at 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 1 librados: starting objecter 2024-03-29T11:35:47.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 1 Processor -- start 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 1 -- start start 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.467+0000 7f64c7ad9640 1 librados: setting wanted keys 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.468+0000 7f64c7ad9640 1 librados: calling monclient init 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.468+0000 7f64c7ad9640 10 monclient: init 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00a9930 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f64c0149dc0 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c5074640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f64c0149dc0 0x7f64c01dea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40544/0 (socket says 172.21.15.45:40544) 2024-03-29T11:35:47.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f64c014ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0070c20 con 0x7f64c014ff40 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c006fb10 con 0x7f64c00a9930 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c006fe30 con 0x7f64c0149dc0 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c7ad9640 10 monclient(hunting): authenticate will time out at 1003.708712s 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.469+0000 7f64c5074640 1 -- 172.21.15.45:0/1327484323 learned_addr learned my addr 172.21.15.45:0/1327484323 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 client.?.objecter ms_handle_connect 0x7f64c0149dc0 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27230140 0 0) 0x7f64c006fe30 con 0x7f64c0149dc0 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:35:47.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.470+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4004eb0 con 0x7f64c0149dc0 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 client.?.objecter ms_handle_connect 0x7f64c00a9930 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163943260 0 0) 0x7f64c006fb10 con 0x7f64c00a9930 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c006fe30 con 0x7f64c00a9930 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645437612 0 0) 0x7f64a4004eb0 con 0x7f64c0149dc0 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c006fb10 con 0x7f64c0149dc0 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b8004380 con 0x7f64c0149dc0 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): dump: 2024-03-29T11:35:47.475 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.471+0000 7f64b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2880092778 0 0) 0x7f64c006fb10 con 0x7f64c0149dc0 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 >> v1:172.21.15.190:6789/0 conn(0x7f64c00a9930 legacy=0x7f64c013b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 >> v1:172.21.15.45:6789/0 conn(0x7f64c014ff40 legacy=0x7f64c01df170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:47.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 1 monclient: found mon.c 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c006ffc0 con 0x7f64c0149dc0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 10 monclient: _renew_subs 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0070730 con 0x7f64c0149dc0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 10 monclient: _renew_subs 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c013f210 con 0x7f64c0149dc0 2024-03-29T11:35:47.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.472+0000 7f64c7ad9640 1 librados: init done 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.473+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b80044e0 con 0x7f64c0149dc0 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.473+0000 7f64b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.473+0000 7f64b7fff640 4 set_mon_vals no callback set 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.474+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b8001430 con 0x7f64c0149dc0 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.474+0000 7f64b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.474+0000 7f64b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.474+0000 7f64b67fc640 10 monclient: dump: 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:47.478 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:47.479 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:47.479 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:47.479 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:47.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f64b8001650 con 0x7f64c0149dc0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64c7ad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64c7ad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f64b8084d20 con 0x7f64c0149dc0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64c7ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 10 client.4376.objecter ms_dispatch 0x7f64c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a8005080 con 0x7f64c0149dc0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64b67fc640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.475+0000 7f64c5875640 10 monclient: get_auth_request con 0x7f64a406da90 auth_method 0 2024-03-29T11:35:47.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.476+0000 7f64b67fc640 10 client.4376.objecter ms_handle_connect 0x7f64a406da90 2024-03-29T11:35:47.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.483+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64b8050600 con 0x7f64c0149dc0 2024-03-29T11:35:47.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.483+0000 7f64b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.483+0000 7f64b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.795+0000 7f64c7ad9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.795+0000 7f64c7ad9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.795+0000 7f64c7ad9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.795+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64a8005330 con 0x7f64c0149dc0 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.796+0000 7f64b67fc640 1 -- 172.21.15.45:0/1327484323 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f64b8052530 con 0x7f64c0149dc0 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.796+0000 7f64b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.796+0000 7f64b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:47.795 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:47.796 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:47.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 10 librados: watch_flush enter 2024-03-29T11:35:47.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 10 librados: watch_flush exit 2024-03-29T11:35:47.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:35:47.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:35:47.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 >> v1:172.21.15.190:6816/1411598462 conn(0x7f64a406da90 legacy=0x7f64a406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 10 monclient: shutdown 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 >> v1:172.21.15.45:6790/0 conn(0x7f64c0149dc0 legacy=0x7f64c01dea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 shutdown_connections 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 >> 172.21.15.45:0/1327484323 conn(0x7f64c00b35e0 msgr2=0x7f64c00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.800+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 shutdown_connections 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.801+0000 7f64c7ad9640 1 -- 172.21.15.45:0/1327484323 wait complete. 2024-03-29T11:35:47.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:47.801+0000 7f64c7ad9640 1 librados: shutdown 2024-03-29T11:35:47.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:54.618 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:54.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.769+0000 7f771e89d640 10 monclient: get_monmap_and_config 2024-03-29T11:35:54.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.769+0000 7f771e89d640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.769+0000 7f771e89d640 10 monclient: monmap: 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:35:54.770950+0000 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:35:54.770950+0000 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.770+0000 7f771e89d640 1 Processor -- start 2024-03-29T11:35:54.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.771+0000 7f771e89d640 1 -- start start 2024-03-29T11:35:54.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.771+0000 7f771e89d640 10 monclient: init 2024-03-29T11:35:54.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.772+0000 7f771e89d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.770 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.772+0000 7f771e89d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-c con 0x7f77100389e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771d89b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f77100389e0 0x7f7710033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:52797/0 (socket says 172.21.15.45:52797) 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771d89b640 1 -- 172.21.15.45:0/602761983 learned_addr learned my addr 172.21.15.45:0/602761983 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-a con 0x7f7710038dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-b con 0x7f7710030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77100c54c0 con 0x7f7710038dc0 2024-03-29T11:35:54.774 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.775+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7710036430 con 0x7f77100389e0 2024-03-29T11:35:54.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.776+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7710039840 con 0x7f7710030660 2024-03-29T11:35:54.775 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.776+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026928539 0 0) 0x7f7710036430 con 0x7f77100389e0 2024-03-29T11:35:54.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771e89d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771e89d640 10 monclient(hunting): authenticate will time out at 1011.016423s 2024-03-29T11:35:54.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704003520 con 0x7f77100389e0 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524288393 0 0) 0x7f7710039840 con 0x7f7710030660 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7710036430 con 0x7f7710030660 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836411831 0 0) 0x7f7704003520 con 0x7f77100389e0 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.777+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 >> v1:172.21.15.45:6790/0 conn(0x7f7710030660 legacy=0x7f77100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 >> v1:172.21.15.45:6789/0 conn(0x7f7710038dc0 legacy=0x7f77100cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 1 monclient: found mon.noname-c 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.778+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710035690 con 0x7f77100389e0 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f7708001690 con 0x7f77100389e0 2024-03-29T11:35:54.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:54.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: dump: 2024-03-29T11:35:54.784 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:54.784 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771e89d640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771e89d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77080019a0 con 0x7f77100389e0 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 1 -- 172.21.15.45:0/602761983 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f7708001d30 con 0x7f77100389e0 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771c899640 10 monclient: dump: 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771e89d640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.779+0000 7f771e89d640 4 set_mon_vals no callback set 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.780+0000 7f771e89d640 10 monclient: shutdown 2024-03-29T11:35:54.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.780+0000 7f771e89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.780+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 >> v1:172.21.15.190:6789/0 conn(0x7f77100389e0 legacy=0x7f7710033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.781+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 shutdown_connections 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.781+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 >> 172.21.15.45:0/602761983 conn(0x7f7710005c00 msgr2=0x7f7710006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.781+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 shutdown_connections 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.781+0000 7f771e89d640 1 -- 172.21.15.45:0/602761983 wait complete. 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 10 monclient: monmap: 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:35:54.787 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 librados: starting msgr at 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 librados: starting objecter 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 Processor -- start 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 -- start start 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 librados: setting wanted keys 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 1 librados: calling monclient init 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.783+0000 7f771e89d640 10 monclient: init 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.784+0000 7f771e89d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.784+0000 7f771e89d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-b con 0x7f7710030660 addr v1:172.21.15.190:6789/0 2024-03-29T11:35:54.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-a con 0x7f77100389e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 10 monclient(hunting): picked mon.noname-c con 0x7f771002f220 addr v1:172.21.15.45:6790/0 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771000a2f0 con 0x7f77100389e0 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771002ffd0 con 0x7f7710030660 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771002fab0 con 0x7f771002f220 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e89d640 10 monclient(hunting): authenticate will time out at 1011.025458s 2024-03-29T11:35:54.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e09c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f771002f220 0x7f771016c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34518/0 (socket says 172.21.15.45:34518) 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.786+0000 7f771e09c640 1 -- 172.21.15.45:0/2236066465 learned_addr learned my addr 172.21.15.45:0/2236066465 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 client.?.objecter ms_handle_connect 0x7f771002f220 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 client.?.objecter ms_handle_connect 0x7f7710030660 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263702985 0 0) 0x7f771002ffd0 con 0x7f7710030660 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76fc004eb0 con 0x7f7710030660 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907936236 0 0) 0x7f771002fab0 con 0x7f771002f220 2024-03-29T11:35:54.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.789+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f771002ffd0 con 0x7f771002f220 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152526434 0 0) 0x7f76fc004eb0 con 0x7f7710030660 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f771002fab0 con 0x7f7710030660 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f77080014b0 con 0x7f7710030660 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.795 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221062179 0 0) 0x7f771002ffd0 con 0x7f771002f220 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.790+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76fc004eb0 con 0x7f771002f220 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f771804fdd0 con 0x7f771002f220 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.796 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70598224 0 0) 0x7f771002fab0 con 0x7f7710030660 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 >> v1:172.21.15.45:6790/0 conn(0x7f771002f220 legacy=0x7f771016c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 >> v1:172.21.15.45:6789/0 conn(0x7f77100389e0 legacy=0x7f7710164100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 monclient: found mon.b 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771002ea20 con 0x7f7710030660 2024-03-29T11:35:54.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 client.?.objecter ms_handle_connect 0x7f77100389e0 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f770e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 10 monclient: _renew_subs 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771002f6d0 con 0x7f7710030660 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 10 monclient: _renew_subs 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.791+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f771000d630 con 0x7f7710030660 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.792+0000 7f771e89d640 1 librados: init done 2024-03-29T11:35:54.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.793+0000 7f771e89d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.793+0000 7f771e89d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.793+0000 7f771e89d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.793+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7710007b90 con 0x7f7710030660 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.793+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7708001810 con 0x7f7710030660 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.794+0000 7f770e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.794+0000 7f770ffff640 4 set_mon_vals no callback set 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.795+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f7708004c70 con 0x7f7710030660 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.795+0000 7f770e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.795+0000 7f770e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.795+0000 7f770e7fc640 10 monclient: dump: 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:35:54.800 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.795+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f770801ac80 con 0x7f7710030660 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f7708084b40 con 0x7f7710030660 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 10 client.4345.objecter ms_dispatch 0x7f7718000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f771d09a640 10 monclient: get_auth_request con 0x7f76fc06de00 auth_method 0 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:54.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:54.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:35:54.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.796+0000 7f770e7fc640 10 client.4345.objecter ms_handle_connect 0x7f76fc06de00 2024-03-29T11:35:54.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.801+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77080502d0 con 0x7f7710030660 2024-03-29T11:35:54.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.801+0000 7f770e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:54.801+0000 7f770e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:55.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.128+0000 7f771e89d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.128+0000 7f771e89d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.128+0000 7f771e89d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:35:55.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.128+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771002fc40 con 0x7f7710030660 2024-03-29T11:35:55.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.132+0000 7f770e7fc640 1 -- 172.21.15.45:0/2236066465 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f77080523d0 con 0x7f7710030660 2024-03-29T11:35:55.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.132+0000 7f770e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:55.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.132+0000 7f770e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:55.134 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:35:55.134 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:55.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 10 librados: watch_flush enter 2024-03-29T11:35:55.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 10 librados: watch_flush exit 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 >> v1:172.21.15.190:6816/1411598462 conn(0x7f76fc06de00 legacy=0x7f76fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 10 monclient: shutdown 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.137+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 >> v1:172.21.15.190:6789/0 conn(0x7f7710030660 legacy=0x7f771000cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.138+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 shutdown_connections 2024-03-29T11:35:55.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.138+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 >> 172.21.15.45:0/2236066465 conn(0x7f7710005c00 msgr2=0x7f7710034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:55.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.138+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 shutdown_connections 2024-03-29T11:35:55.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.138+0000 7f771e89d640 1 -- 172.21.15.45:0/2236066465 wait complete. 2024-03-29T11:35:55.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:35:55.138+0000 7f771e89d640 1 librados: shutdown 2024-03-29T11:35:55.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:01.961 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:02.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.109+0000 7fdad094c640 10 monclient: get_monmap_and_config 2024-03-29T11:36:02.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.109+0000 7fdad094c640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.109+0000 7fdad094c640 10 monclient: monmap: 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:02.110978+0000 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:02.110978+0000 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:02.109 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.111+0000 7fdad094c640 1 Processor -- start 2024-03-29T11:36:02.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.112+0000 7fdad094c640 1 -- start start 2024-03-29T11:36:02.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.112+0000 7fdad094c640 10 monclient: init 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc0b2800 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc0b3220 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc1439c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0b3840 con 0x7fdacc0b3220 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc06f230 con 0x7fdacc1439c0 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc06ef50 con 0x7fdacc0b2800 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.113+0000 7fdad094c640 10 monclient(hunting): authenticate will time out at 1018.352684s 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.116+0000 7fdaca59c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdacc0b3220 0x7fdacc146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45561/0 (socket says 172.21.15.45:45561) 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.116+0000 7fdaca59c640 1 -- 172.21.15.45:0/495645661 learned_addr learned my addr 172.21.15.45:0/495645661 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250835401 0 0) 0x7fdacc0b3840 con 0x7fdacc0b3220 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab4003520 con 0x7fdacc0b3220 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452938270 0 0) 0x7fdab4003520 con 0x7fdacc0b3220 2024-03-29T11:36:02.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 >> v1:172.21.15.45:6790/0 conn(0x7fdacc0b2800 legacy=0x7fdacc1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.117+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 >> v1:172.21.15.190:6789/0 conn(0x7fdacc1439c0 legacy=0x7fdacc151c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 1 monclient: found mon.noname-a 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:36:02.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc071d30 con 0x7fdacc0b3220 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdac0001670 con 0x7fdacc0b3220 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdac9d9b640 10 monclient: dump: 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.120 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:02.121 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:02.121 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:02.121 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdad094c640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:36:02.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.120+0000 7fdad094c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac00018d0 con 0x7fdacc0b3220 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 1 -- 172.21.15.45:0/495645661 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdac0001d20 con 0x7fdacc0b3220 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdac9d9b640 10 monclient: dump: 2024-03-29T11:36:02.122 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdad094c640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:02.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.123+0000 7fdad094c640 4 set_mon_vals no callback set 2024-03-29T11:36:02.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.124+0000 7fdad094c640 10 monclient: shutdown 2024-03-29T11:36:02.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.124+0000 7fdad094c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:02.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.124+0000 7fdad094c640 1 -- 172.21.15.45:0/495645661 >> v1:172.21.15.45:6789/0 conn(0x7fdacc0b3220 legacy=0x7fdacc146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.129+0000 7fdad094c640 1 -- 172.21.15.45:0/495645661 shutdown_connections 2024-03-29T11:36:02.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.129+0000 7fdad094c640 1 -- 172.21.15.45:0/495645661 >> 172.21.15.45:0/495645661 conn(0x7fdacc066a00 msgr2=0x7fdacc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.131+0000 7fdad094c640 1 -- 172.21.15.45:0/495645661 shutdown_connections 2024-03-29T11:36:02.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.131+0000 7fdad094c640 1 -- 172.21.15.45:0/495645661 wait complete. 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.132+0000 7fdad094c640 10 monclient: build_initial_monmap 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.132+0000 7fdad094c640 10 monclient: monmap: 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:02.132 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:02.133 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.133+0000 7fdad094c640 1 librados: starting msgr at 2024-03-29T11:36:02.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.133+0000 7fdad094c640 1 librados: starting objecter 2024-03-29T11:36:02.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.133+0000 7fdad094c640 1 Processor -- start 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.134+0000 7fdad094c640 1 -- start start 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.134+0000 7fdad094c640 1 librados: setting wanted keys 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.134+0000 7fdad094c640 1 librados: calling monclient init 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.134+0000 7fdad094c640 10 monclient: init 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.135+0000 7fdad094c640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:02.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.135+0000 7fdad094c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:02.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.135+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc0b2800 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:02.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc0b3220 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:02.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc1439c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:02.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc149e50 con 0x7fdacc1439c0 2024-03-29T11:36:02.136 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc14a0c0 con 0x7fdacc0b2800 2024-03-29T11:36:02.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc1490f0 con 0x7fdacc0b3220 2024-03-29T11:36:02.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:02.137 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.137+0000 7fdad094c640 10 monclient(hunting): authenticate will time out at 1018.376920s 2024-03-29T11:36:02.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.141+0000 7fdacad9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdacc0b2800 0x7fdacc13c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:51269/0 (socket says 172.21.15.45:51269) 2024-03-29T11:36:02.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.141+0000 7fdaca59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdacc0b3220 0x7fdacc0b5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48725/0 (socket says 172.21.15.45:48725) 2024-03-29T11:36:02.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.141+0000 7fdaca59c640 1 -- 172.21.15.45:0/1280863630 learned_addr learned my addr 172.21.15.45:0/1280863630 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:02.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.142+0000 7fdab37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc0b3220 2024-03-29T11:36:02.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.142+0000 7fdab37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259713118 0 0) 0x7fdacc1490f0 con 0x7fdacc0b3220 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:36:02.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.143+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8004eb0 con 0x7fdacc0b3220 2024-03-29T11:36:02.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc1439c0 2024-03-29T11:36:02.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc0b2800 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799941472 0 0) 0x7fdacc14a0c0 con 0x7fdacc0b2800 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc1490f0 con 0x7fdacc0b2800 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.144+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334266547 0 0) 0x7fdab8004eb0 con 0x7fdacc0b3220 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdacc14a0c0 con 0x7fdacc0b3220 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdac0001d20 con 0x7fdacc0b3220 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:02.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): dump: 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619026216 0 0) 0x7fdacc1490f0 con 0x7fdacc0b2800 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdab8004eb0 con 0x7fdacc0b2800 2024-03-29T11:36:02.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdabc001750 con 0x7fdacc0b2800 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): dump: 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962666224 0 0) 0x7fdacc149e50 con 0x7fdacc1439c0 2024-03-29T11:36:02.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc1490f0 con 0x7fdacc1439c0 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3337355975 0 0) 0x7fdacc14a0c0 con 0x7fdacc0b3220 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 >> v1:172.21.15.190:6789/0 conn(0x7fdacc0b2800 legacy=0x7fdacc13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 >> v1:172.21.15.45:6789/0 conn(0x7fdacc1439c0 legacy=0x7fdacc0b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.145+0000 7fdab37fe640 1 monclient: found mon.c 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc1486d0 con 0x7fdacc0b3220 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:02.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac0004d20 con 0x7fdacc0b3220 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdac0005030 con 0x7fdacc0b3220 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdab37fe640 10 monclient: dump: 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:02.149 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:02.150 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:02.150 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.146+0000 7fdac8d99640 4 set_mon_vals no callback set 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.148+0000 7fdad094c640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.148+0000 7fdad094c640 10 monclient: _renew_subs 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.148+0000 7fdad094c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.148+0000 7fdad094c640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc148980 con 0x7fdacc0b3220 2024-03-29T11:36:02.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.149+0000 7fdad094c640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.149+0000 7fdad094c640 10 monclient: _renew_subs 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.149+0000 7fdad094c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.149+0000 7fdad094c640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc13c910 con 0x7fdacc0b3220 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.149+0000 7fdad094c640 1 librados: init done 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.150+0000 7fdab17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.150+0000 7fdab17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.150+0000 7fdab17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.150+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa4005080 con 0x7fdacc0b3220 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.151+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fdac001b000 con 0x7fdacc0b3220 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.152+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fdac0084e40 con 0x7fdacc0b3220 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.152+0000 7fdab37fe640 10 client.4385.objecter ms_dispatch 0x7fdacc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.152+0000 7fdab37fe640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.152+0000 7fdab37fe640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:02.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.152+0000 7fdab37fe640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:36:02.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.153+0000 7fdacad9d640 10 monclient: get_auth_request con 0x7fdab806dfe0 auth_method 0 2024-03-29T11:36:02.153 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.154+0000 7fdab37fe640 10 client.4385.objecter ms_handle_connect 0x7fdab806dfe0 2024-03-29T11:36:02.159 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.160+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdac0050720 con 0x7fdacc0b3220 2024-03-29T11:36:02.159 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.160+0000 7fdab37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.159 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.160+0000 7fdab37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:02.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.492+0000 7fdab17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.492+0000 7fdab17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.492+0000 7fdab17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:02.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.492+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaa4005330 con 0x7fdacc0b3220 2024-03-29T11:36:02.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.493+0000 7fdab37fe640 1 -- 172.21.15.45:0/1280863630 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fdac0052650 con 0x7fdacc0b3220 2024-03-29T11:36:02.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.493+0000 7fdab37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.493+0000 7fdab37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:02.492 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:02.493 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:02.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 10 librados: watch_flush enter 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 10 librados: watch_flush exit 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 >> v1:172.21.15.190:6816/1411598462 conn(0x7fdab806dfe0 legacy=0x7fdab80704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 10 monclient: shutdown 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:02.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.497+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 >> v1:172.21.15.45:6790/0 conn(0x7fdacc0b3220 legacy=0x7fdacc0b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.501+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 shutdown_connections 2024-03-29T11:36:02.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.501+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 >> 172.21.15.45:0/1280863630 conn(0x7fdacc066a00 msgr2=0x7fdacc141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.501+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 shutdown_connections 2024-03-29T11:36:02.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.504+0000 7fdab17fa640 1 -- 172.21.15.45:0/1280863630 wait complete. 2024-03-29T11:36:02.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:02.504+0000 7fdab17fa640 1 librados: shutdown 2024-03-29T11:36:02.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:09.329 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.489+0000 7fa64e8f2640 10 monclient: get_monmap_and_config 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.489+0000 7fa64e8f2640 10 monclient: build_initial_monmap 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.489+0000 7fa64e8f2640 10 monclient: monmap: 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.487 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:09.489465+0000 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:09.489465+0000 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:09.488 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.490+0000 7fa64e8f2640 1 Processor -- start 2024-03-29T11:36:09.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.490+0000 7fa64e8f2640 1 -- start start 2024-03-29T11:36:09.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.490+0000 7fa64e8f2640 10 monclient: init 2024-03-29T11:36:09.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-a con 0x7fa6400389e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-b con 0x7fa640038dc0 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-c con 0x7fa640030660 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400c54c0 con 0x7fa6400389e0 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa640036430 con 0x7fa640030660 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa640039840 con 0x7fa640038dc0 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:09.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.491+0000 7fa64e8f2640 10 monclient(hunting): authenticate will time out at 1025.729835s 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.497+0000 7fa64d8f0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fa6400389e0 0x7fa640033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42365/0 (socket says 172.21.15.45:42365) 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.498+0000 7fa64d8f0640 1 -- 172.21.15.45:0/629911824 learned_addr learned my addr 172.21.15.45:0/629911824 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386426583 0 0) 0x7fa640036430 con 0x7fa640030660 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa634003520 con 0x7fa640030660 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243046120 0 0) 0x7fa634003520 con 0x7fa640030660 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 >> v1:172.21.15.45:6790/0 conn(0x7fa640038dc0 legacy=0x7fa6400cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 >> v1:172.21.15.45:6789/0 conn(0x7fa6400389e0 legacy=0x7fa640033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: _finish_hunting 0 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 monclient: found mon.noname-c 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640035690 con 0x7fa640030660 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: _finish_auth 0 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: _check_auth_tickets 2024-03-29T11:36:09.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa644001690 con 0x7fa640030660 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.499+0000 7fa64c8ee640 10 monclient: dump: 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:09.501 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64e8f2640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64e8f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa644001860 con 0x7fa640030660 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 1 -- 172.21.15.45:0/629911824 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa644001cb0 con 0x7fa640030660 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64c8ee640 10 monclient: dump: 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.502 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64e8f2640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.500+0000 7fa64e8f2640 4 set_mon_vals no callback set 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.501+0000 7fa64e8f2640 10 monclient: shutdown 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.501+0000 7fa64e8f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.501+0000 7fa64e8f2640 1 -- 172.21.15.45:0/629911824 >> v1:172.21.15.190:6789/0 conn(0x7fa640030660 legacy=0x7fa6400d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.503+0000 7fa64e8f2640 1 -- 172.21.15.45:0/629911824 shutdown_connections 2024-03-29T11:36:09.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.503+0000 7fa64e8f2640 1 -- 172.21.15.45:0/629911824 >> 172.21.15.45:0/629911824 conn(0x7fa640005c00 msgr2=0x7fa640006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.503+0000 7fa64e8f2640 1 -- 172.21.15.45:0/629911824 shutdown_connections 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.503+0000 7fa64e8f2640 1 -- 172.21.15.45:0/629911824 wait complete. 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.504+0000 7fa64e8f2640 10 monclient: build_initial_monmap 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.504+0000 7fa64e8f2640 10 monclient: monmap: 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.506 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 librados: starting msgr at 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 librados: starting objecter 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 Processor -- start 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 -- start start 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 librados: setting wanted keys 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 1 librados: calling monclient init 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.505+0000 7fa64e8f2640 10 monclient: init 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-b con 0x7fa640030660 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:09.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-c con 0x7fa6400389e0 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient(hunting): picked mon.noname-a con 0x7fa6400c29c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64002fe50 con 0x7fa6400c29c0 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400c58f0 con 0x7fa640030660 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa640169c40 con 0x7fa6400389e0 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.506+0000 7fa64e8f2640 10 monclient(hunting): authenticate will time out at 1025.744348s 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.509+0000 7fa64d0ef640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa6400389e0 0x7fa6400c22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36487/0 (socket says 172.21.15.45:36487) 2024-03-29T11:36:09.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.509+0000 7fa64d0ef640 1 -- 172.21.15.45:0/465510944 learned_addr learned my addr 172.21.15.45:0/465510944 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter ms_handle_connect 0x7fa640030660 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6400389e0 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6400c29c0 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227196386 0 0) 0x7fa6400c58f0 con 0x7fa640030660 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.512+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c004eb0 con 0x7fa640030660 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752967490 0 0) 0x7fa62c004eb0 con 0x7fa640030660 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6400c58f0 con 0x7fa640030660 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa64804fdd0 con 0x7fa640030660 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 10 monclient(hunting): dump: 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.519 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.513+0000 7fa63e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:09.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958204523 0 0) 0x7fa64002fe50 con 0x7fa6400c29c0 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c005680 con 0x7fa6400c29c0 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3982298847 0 0) 0x7fa62c005680 con 0x7fa6400c29c0 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64002fe50 con 0x7fa6400c29c0 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa644001b70 con 0x7fa6400c29c0 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.521 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient(hunting): dump: 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.522 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2720105425 0 0) 0x7fa64002fe50 con 0x7fa6400c29c0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 >> v1:172.21.15.45:6790/0 conn(0x7fa6400389e0 legacy=0x7fa6400c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 >> v1:172.21.15.190:6789/0 conn(0x7fa640030660 legacy=0x7fa64002d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 monclient: found mon.a 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64002d860 con 0x7fa6400c29c0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa63e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa64e8f2640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa64e8f2640 10 monclient: _renew_subs 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa64e8f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.514+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64016a160 con 0x7fa6400c29c0 2024-03-29T11:36:09.523 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.515+0000 7fa64e8f2640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.515+0000 7fa64e8f2640 10 monclient: _renew_subs 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.515+0000 7fa64e8f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.515+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6400321e0 con 0x7fa6400c29c0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.515+0000 7fa64e8f2640 1 librados: init done 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.516+0000 7fa64e8f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.516+0000 7fa64e8f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.516+0000 7fa64e8f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.516+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa640007b90 con 0x7fa6400c29c0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa644001d50 con 0x7fa6400c29c0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa644001b70 con 0x7fa6400c29c0 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:09.524 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 10 monclient: dump: 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63f7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.518+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fa6440197b0 con 0x7fa6400c29c0 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.519+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fa6440834d0 con 0x7fa6400c29c0 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.519+0000 7fa63e7fc640 10 client.4374.objecter ms_dispatch 0x7fa648000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:09.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.519+0000 7fa63e7fc640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.519+0000 7fa63e7fc640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.519+0000 7fa63e7fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.520+0000 7fa64d8f0640 10 monclient: get_auth_request con 0x7fa62c06e310 auth_method 0 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.520+0000 7fa63e7fc640 10 client.4374.objecter ms_handle_connect 0x7fa62c06e310 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.526+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa64404eee0 con 0x7fa6400c29c0 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.526+0000 7fa63e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.526+0000 7fa63e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:09.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.869+0000 7fa64e8f2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.869+0000 7fa64e8f2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.869+0000 7fa64e8f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:09.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.869+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64016a4b0 con 0x7fa6400c29c0 2024-03-29T11:36:09.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.871+0000 7fa63e7fc640 1 -- 172.21.15.45:0/465510944 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fa644050e10 con 0x7fa6400c29c0 2024-03-29T11:36:09.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.871+0000 7fa63e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.871+0000 7fa63e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:09.870 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:09.870 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.874+0000 7fa64e8f2640 10 librados: watch_flush enter 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 10 librados: watch_flush exit 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 >> v1:172.21.15.190:6816/1411598462 conn(0x7fa62c06e310 legacy=0x7fa62c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 10 monclient: shutdown 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 >> v1:172.21.15.45:6789/0 conn(0x7fa6400c29c0 legacy=0x7fa640174db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 shutdown_connections 2024-03-29T11:36:09.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 >> 172.21.15.45:0/465510944 conn(0x7fa640005c00 msgr2=0x7fa64003ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 shutdown_connections 2024-03-29T11:36:09.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 -- 172.21.15.45:0/465510944 wait complete. 2024-03-29T11:36:09.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:09.875+0000 7fa64e8f2640 1 librados: shutdown 2024-03-29T11:36:09.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:16.700 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:16.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.853+0000 7fae21d10640 10 monclient: get_monmap_and_config 2024-03-29T11:36:16.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.853+0000 7fae21d10640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.853+0000 7fae21d10640 10 monclient: monmap: 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:16.854745+0000 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:16.854745+0000 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:16.853 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.854+0000 7fae21d10640 1 Processor -- start 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.854+0000 7fae21d10640 1 -- start start 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.854+0000 7fae21d10640 10 monclient: init 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.855+0000 7fae21d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.855+0000 7fae21d10640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.855+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-a con 0x7fae1c0af390 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:16.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.855+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-c con 0x7fae1c0b3990 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:16.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae20d0e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fae1c0af390 0x7fae1c1459f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54613/0 (socket says 172.21.15.45:54613) 2024-03-29T11:36:16.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-b con 0x7fae1c0b16d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:16.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0b47f0 con 0x7fae1c0af390 2024-03-29T11:36:16.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0b1b20 con 0x7fae1c0b3990 2024-03-29T11:36:16.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0b44a0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae21d10640 10 monclient(hunting): authenticate will time out at 1033.100740s 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.861+0000 7fae20d0e640 1 -- 172.21.15.45:0/1094879025 learned_addr learned my addr 172.21.15.45:0/1094879025 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676929608 0 0) 0x7fae1c0b44a0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae04003520 con 0x7fae1c0b16d0 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76347064 0 0) 0x7fae04003520 con 0x7fae1c0b16d0 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 >> v1:172.21.15.190:6789/0 conn(0x7fae1c0b3990 legacy=0x7fae1c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 >> v1:172.21.15.45:6789/0 conn(0x7fae1c0af390 legacy=0x7fae1c1459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:16.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0b4c80 con 0x7fae1c0b16d0 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae14001a10 con 0x7fae1c0b16d0 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae1b7fe640 10 monclient: dump: 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.864 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae21d10640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.862+0000 7fae21d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae14001ca0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 1 -- 172.21.15.45:0/1094879025 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae14002030 con 0x7fae1c0b16d0 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:16.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae1b7fe640 10 monclient: dump: 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae21d10640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:16.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.863+0000 7fae21d10640 4 set_mon_vals no callback set 2024-03-29T11:36:16.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.864+0000 7fae21d10640 10 monclient: shutdown 2024-03-29T11:36:16.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.864+0000 7fae21d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.864+0000 7fae21d10640 1 -- 172.21.15.45:0/1094879025 >> v1:172.21.15.45:6790/0 conn(0x7fae1c0b16d0 legacy=0x7fae1c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.867+0000 7fae21d10640 1 -- 172.21.15.45:0/1094879025 shutdown_connections 2024-03-29T11:36:16.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.867+0000 7fae21d10640 1 -- 172.21.15.45:0/1094879025 >> 172.21.15.45:0/1094879025 conn(0x7fae1c06d230 msgr2=0x7fae1c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.867+0000 7fae21d10640 1 -- 172.21.15.45:0/1094879025 shutdown_connections 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.867+0000 7fae21d10640 1 -- 172.21.15.45:0/1094879025 wait complete. 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.868+0000 7fae21d10640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.868+0000 7fae21d10640 10 monclient: monmap: 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.869 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 librados: starting msgr at 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 librados: starting objecter 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 Processor -- start 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 -- start start 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 librados: setting wanted keys 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 1 librados: calling monclient init 2024-03-29T11:36:16.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.869+0000 7fae21d10640 10 monclient: init 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.871+0000 7fae21d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.871+0000 7fae21d10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.871+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-a con 0x7fae1c0af390 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.871+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-c con 0x7fae1c0b16d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 10 monclient(hunting): picked mon.noname-b con 0x7fae1c0b3990 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c1e73e0 con 0x7fae1c0af390 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0aa190 con 0x7fae1c0b3990 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c1e6f00 con 0x7fae1c0b16d0 2024-03-29T11:36:16.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.873+0000 7fae21d10640 10 monclient(hunting): authenticate will time out at 1033.112189s 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.876+0000 7fae1bfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fae1c0b16d0 0x7fae1c1e7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58479/0 (socket says 172.21.15.45:58479) 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.876+0000 7fae1bfff640 1 -- 172.21.15.45:0/133078708 learned_addr learned my addr 172.21.15.45:0/133078708 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.876+0000 7fae197fa640 10 client.?.objecter ms_handle_connect 0x7fae1c0b16d0 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.876+0000 7fae197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484167680 0 0) 0x7fae1c1e6f00 con 0x7fae1c0b16d0 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08004eb0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 client.?.objecter ms_handle_connect 0x7fae1c0af390 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 client.?.objecter ms_handle_connect 0x7fae1c0b3990 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522689276 0 0) 0x7fae1c1e73e0 con 0x7fae1c0af390 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.877+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae080051d0 con 0x7fae1c0af390 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470982381 0 0) 0x7fae1c0aa190 con 0x7fae1c0b3990 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.878+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08005510 con 0x7fae1c0b3990 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776596818 0 0) 0x7fae08004eb0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae08005980 con 0x7fae1c0b16d0 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3532546656 0 0) 0x7fae08005510 con 0x7fae1c0b3990 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae08004eb0 con 0x7fae1c0b3990 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3598277259 0 0) 0x7fae080051d0 con 0x7fae1c0af390 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae08005510 con 0x7fae1c0af390 2024-03-29T11:36:16.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae10003fa0 con 0x7fae1c0b16d0 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): dump: 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:16.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae140018d0 con 0x7fae1c0b3990 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): dump: 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae0c001490 con 0x7fae1c0af390 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.879+0000 7fae197fa640 10 monclient(hunting): dump: 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2188778909 0 0) 0x7fae08004eb0 con 0x7fae1c0b3990 2024-03-29T11:36:16.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 >> v1:172.21.15.45:6790/0 conn(0x7fae1c0b16d0 legacy=0x7fae1c1e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 >> v1:172.21.15.45:6789/0 conn(0x7fae1c0af390 legacy=0x7fae1c06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 1 monclient: found mon.b 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c1e7090 con 0x7fae1c0b3990 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 10 monclient: _renew_subs 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c0ad8f0 con 0x7fae1c0b3990 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:16.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 10 monclient: _renew_subs 2024-03-29T11:36:16.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:36:16.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c149ee0 con 0x7fae1c0b3990 2024-03-29T11:36:16.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.882+0000 7fae21d10640 1 librados: init done 2024-03-29T11:36:16.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae14001a30 con 0x7fae1c0b3990 2024-03-29T11:36:16.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fae140015d0 con 0x7fae1c0b3990 2024-03-29T11:36:16.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae197fa640 10 monclient: dump: 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.883+0000 7fae1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.885+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fae14019cf0 con 0x7fae1c0b3990 2024-03-29T11:36:16.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.886+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fae14053680 con 0x7fae1c0b3990 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.886+0000 7fae197fa640 10 client.4357.objecter ms_dispatch 0x7fae1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.886+0000 7fae197fa640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.886+0000 7fae197fa640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.886+0000 7fae197fa640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.887+0000 7fae20d0e640 10 monclient: get_auth_request con 0x7fae0806ea00 auth_method 0 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.888+0000 7fae21d10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.888+0000 7fae21d10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.888+0000 7fae21d10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.888+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc005080 con 0x7fae1c0b3990 2024-03-29T11:36:16.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.891+0000 7fae197fa640 10 client.4357.objecter ms_handle_connect 0x7fae0806ea00 2024-03-29T11:36:16.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.895+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae1404f920 con 0x7fae1c0b3990 2024-03-29T11:36:16.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.895+0000 7fae197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:16.895+0000 7fae197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:17.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.223+0000 7fae21d10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.223+0000 7fae21d10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.223+0000 7fae21d10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:36:17.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.223+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadfc005330 con 0x7fae1c0b3990 2024-03-29T11:36:17.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.226+0000 7fae197fa640 1 -- 172.21.15.45:0/133078708 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fae1404f920 con 0x7fae1c0b3990 2024-03-29T11:36:17.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.226+0000 7fae197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.226+0000 7fae197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:17.225 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:17.225 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 10 librados: watch_flush enter 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 10 librados: watch_flush exit 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 10 client.4357.objecter successfully canceled tick 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 >> v1:172.21.15.190:6816/1411598462 conn(0x7fae0806ea00 legacy=0x7fae08070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 10 monclient: shutdown 2024-03-29T11:36:17.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 >> v1:172.21.15.190:6789/0 conn(0x7fae1c0b3990 legacy=0x7fae1c06f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 shutdown_connections 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.230+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 >> 172.21.15.45:0/133078708 conn(0x7fae1c06d230 msgr2=0x7fae1c142910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.231+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 shutdown_connections 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.231+0000 7fae21d10640 1 -- 172.21.15.45:0/133078708 wait complete. 2024-03-29T11:36:17.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:17.231+0000 7fae21d10640 1 librados: shutdown 2024-03-29T11:36:17.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:24.054 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:24.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.204+0000 7f592b53d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:24.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.204+0000 7f592b53d640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.204+0000 7f592b53d640 10 monclient: monmap: 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:24.205798+0000 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:24.205798+0000 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.205+0000 7f592b53d640 1 Processor -- start 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 1 -- start start 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient: init 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-a con 0x7f59240a9ac0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-c con 0x7f592413ee90 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-b con 0x7f59240af900 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.206+0000 7f592b53d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240afd10 con 0x7f59240a9ac0 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.207+0000 7f592b53d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240af6f0 con 0x7f592413ee90 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.207+0000 7f592b53d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240aac80 con 0x7f59240af900 2024-03-29T11:36:24.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.207+0000 7f592b53d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.207+0000 7f592b53d640 10 monclient(hunting): authenticate will time out at 1040.446129s 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f5928ad8640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f592413ee90 0x7f592413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:39295/0 (socket says 172.21.15.45:39295) 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f5928ad8640 1 -- 172.21.15.45:0/1746689292 learned_addr learned my addr 172.21.15.45:0/1746689292 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693822907 0 0) 0x7f59240af6f0 con 0x7f592413ee90 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910003520 con 0x7f592413ee90 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879140069 0 0) 0x7f5910003520 con 0x7f592413ee90 2024-03-29T11:36:24.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.210+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 >> v1:172.21.15.45:6790/0 conn(0x7f59240af900 legacy=0x7f5924149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:24.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 >> v1:172.21.15.45:6789/0 conn(0x7f59240a9ac0 legacy=0x7f59240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 1 monclient: found mon.noname-c 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240aaf60 con 0x7f592413ee90 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f591c001a10 con 0x7f592413ee90 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f591bfff640 10 monclient: dump: 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.216 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f592b53d640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.215+0000 7f592b53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f591c001be0 con 0x7f592413ee90 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 1 -- 172.21.15.45:0/1746689292 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f591c002030 con 0x7f592413ee90 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:24.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f591bfff640 10 monclient: dump: 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:24.218 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f592b53d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:24.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.216+0000 7f592b53d640 4 set_mon_vals no callback set 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.217+0000 7f592b53d640 10 monclient: shutdown 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.217+0000 7f592b53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.217+0000 7f592b53d640 1 -- 172.21.15.45:0/1746689292 >> v1:172.21.15.190:6789/0 conn(0x7f592413ee90 legacy=0x7f592413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.219+0000 7f592b53d640 1 -- 172.21.15.45:0/1746689292 shutdown_connections 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.219+0000 7f592b53d640 1 -- 172.21.15.45:0/1746689292 >> 172.21.15.45:0/1746689292 conn(0x7f59240b35e0 msgr2=0x7f59240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.220+0000 7f592b53d640 1 -- 172.21.15.45:0/1746689292 shutdown_connections 2024-03-29T11:36:24.220 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.220+0000 7f592b53d640 1 -- 172.21.15.45:0/1746689292 wait complete. 2024-03-29T11:36:24.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.221+0000 7f592b53d640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.221+0000 7f592b53d640 10 monclient: monmap: 2024-03-29T11:36:24.221 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:24.221 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.221+0000 7f592b53d640 1 librados: starting msgr at 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.221+0000 7f592b53d640 1 librados: starting objecter 2024-03-29T11:36:24.222 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.221+0000 7f592b53d640 1 Processor -- start 2024-03-29T11:36:24.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 -- start start 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 librados: setting wanted keys 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 librados: calling monclient init 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient: init 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-a con 0x7f59240a9ac0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-c con 0x7f59240af900 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient(hunting): picked mon.noname-b con 0x7f592413ee90 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5924141150 con 0x7f59240a9ac0 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5924140e20 con 0x7f592413ee90 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5924140a00 con 0x7f59240af900 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.222+0000 7f592b53d640 10 monclient(hunting): authenticate will time out at 1040.461919s 2024-03-29T11:36:24.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.225+0000 7f5928ad8640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f59240af900 0x7f5924139330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:32903/0 (socket says 172.21.15.45:32903) 2024-03-29T11:36:24.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.225+0000 7f5928ad8640 1 -- 172.21.15.45:0/1837855506 learned_addr learned my addr 172.21.15.45:0/1837855506 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:24.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 client.?.objecter ms_handle_connect 0x7f59240af900 2024-03-29T11:36:24.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607988955 0 0) 0x7f5924140a00 con 0x7f59240af900 2024-03-29T11:36:24.232 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5908004eb0 con 0x7f59240af900 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.230+0000 7f5919ffb640 10 client.?.objecter ms_handle_connect 0x7f59240a9ac0 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 30857625 0 0) 0x7f5908004eb0 con 0x7f59240af900 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5924140a00 con 0x7f59240af900 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f591c001fb0 con 0x7f59240af900 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:24.233 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2049594723 0 0) 0x7f5924140a00 con 0x7f59240af900 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 >> v1:172.21.15.190:6789/0 conn(0x7f592413ee90 legacy=0x7f59241415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 >> v1:172.21.15.45:6789/0 conn(0x7f59240a9ac0 legacy=0x7f59240b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.231+0000 7f5919ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 1 monclient: found mon.c 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.234 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924139a40 con 0x7f59240af900 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f5919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 10 monclient: _renew_subs 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5924139cf0 con 0x7f59240af900 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 10 monclient: _renew_subs 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59240b2540 con 0x7f59240af900 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.232+0000 7f592b53d640 1 librados: init done 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.233+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f591c004e10 con 0x7f59240af900 2024-03-29T11:36:24.235 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.233+0000 7f5919ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.233+0000 7f591b7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f592b53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f591c0051e0 con 0x7f59240af900 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f5919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f5919ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f5919ffb640 10 monclient: dump: 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:24.237 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f592b53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f592b53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.234+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5904005080 con 0x7f59240af900 2024-03-29T11:36:24.238 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.239+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f591c01b1b0 con 0x7f59240af900 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f591c084f40 con 0x7f59240af900 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f5919ffb640 10 client.4397.objecter ms_dispatch 0x7f5924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f59292d9640 10 monclient: get_auth_request con 0x7f590806dc10 auth_method 0 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f5919ffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f5919ffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.240+0000 7f5919ffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T11:36:24.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.241+0000 7f5919ffb640 10 client.4397.objecter ms_handle_connect 0x7f590806dc10 2024-03-29T11:36:24.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.244+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f591c050650 con 0x7f59240af900 2024-03-29T11:36:24.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.244+0000 7f5919ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.244+0000 7f5919ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:24.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.564+0000 7f592b53d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.564+0000 7f592b53d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.564+0000 7f592b53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:24.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.564+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5904005330 con 0x7f59240af900 2024-03-29T11:36:24.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.567+0000 7f5919ffb640 1 -- 172.21.15.45:0/1837855506 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f591c052750 con 0x7f59240af900 2024-03-29T11:36:24.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.567+0000 7f5919ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.567+0000 7f5919ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:24.567 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:24.567 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:24.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.572+0000 7f592b53d640 10 librados: watch_flush enter 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 10 librados: watch_flush exit 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 10 client.4397.objecter successfully canceled tick 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 >> v1:172.21.15.190:6816/1411598462 conn(0x7f590806dc10 legacy=0x7f59080700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 10 monclient: shutdown 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 >> v1:172.21.15.45:6790/0 conn(0x7f59240af900 legacy=0x7f5924139330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 shutdown_connections 2024-03-29T11:36:24.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.573+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 >> 172.21.15.45:0/1837855506 conn(0x7f59240b35e0 msgr2=0x7f592413b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.574+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 shutdown_connections 2024-03-29T11:36:24.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.574+0000 7f592b53d640 1 -- 172.21.15.45:0/1837855506 wait complete. 2024-03-29T11:36:24.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:24.574+0000 7f592b53d640 1 librados: shutdown 2024-03-29T11:36:24.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:31.395 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.534+0000 7fa5f4216640 10 monclient: get_monmap_and_config 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.534+0000 7fa5f4216640 10 monclient: build_initial_monmap 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.534+0000 7fa5f4216640 10 monclient: monmap: 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:31.536125+0000 2024-03-29T11:36:31.534 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:31.536125+0000 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:31.535 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.535+0000 7fa5f4216640 1 Processor -- start 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.536+0000 7fa5f4216640 1 -- start start 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.536+0000 7fa5f4216640 10 monclient: init 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.536+0000 7fa5f4216640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.536+0000 7fa5f4216640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0aaef0 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec145bd0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec145eb0 con 0x7fa5ec13f650 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:31.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.537+0000 7fa5f4216640 10 monclient(hunting): authenticate will time out at 1047.776221s 2024-03-29T11:36:31.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f17b1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa5ec13f650 0x7fa5ec0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55824/0 (socket says 172.21.15.45:55824) 2024-03-29T11:36:31.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f17b1640 1 -- 172.21.15.45:0/1629426982 learned_addr learned my addr 172.21.15.45:0/1629426982 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068305272 0 0) 0x7fa5ec145eb0 con 0x7fa5ec13f650 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d8003520 con 0x7fa5ec13f650 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464752175 0 0) 0x7fa5ec0aaef0 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ec145eb0 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882145968 0 0) 0x7fa5d8003520 con 0x7fa5ec13f650 2024-03-29T11:36:31.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.538+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 >> v1:172.21.15.190:6789/0 conn(0x7fa5ec1457f0 legacy=0x7fa5ec13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 >> v1:172.21.15.45:6789/0 conn(0x7fa5ec0a9d70 legacy=0x7fa5ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: _finish_hunting 0 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 1 monclient: found mon.noname-b 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec146340 con 0x7fa5ec13f650 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: _finish_auth 0 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: _check_auth_tickets 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa5dc001690 con 0x7fa5ec13f650 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f0fb0640 10 monclient: dump: 2024-03-29T11:36:31.541 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f4216640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.539+0000 7fa5f4216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5dc0018b0 con 0x7fa5ec13f650 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:31.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 1 -- 172.21.15.45:0/1629426982 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa5dc001c40 con 0x7fa5ec13f650 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f0fb0640 10 monclient: dump: 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f4216640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:31.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.540+0000 7fa5f4216640 4 set_mon_vals no callback set 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.541+0000 7fa5f4216640 10 monclient: shutdown 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.541+0000 7fa5f4216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.541+0000 7fa5f4216640 1 -- 172.21.15.45:0/1629426982 >> v1:172.21.15.45:6790/0 conn(0x7fa5ec13f650 legacy=0x7fa5ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.541+0000 7fa5f4216640 1 -- 172.21.15.45:0/1629426982 shutdown_connections 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.541+0000 7fa5f4216640 1 -- 172.21.15.45:0/1629426982 >> 172.21.15.45:0/1629426982 conn(0x7fa5ec066a00 msgr2=0x7fa5ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.542+0000 7fa5f4216640 1 -- 172.21.15.45:0/1629426982 shutdown_connections 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.542+0000 7fa5f4216640 1 -- 172.21.15.45:0/1629426982 wait complete. 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.542+0000 7fa5f4216640 10 monclient: build_initial_monmap 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.542+0000 7fa5f4216640 10 monclient: monmap: 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:31.546 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 librados: starting msgr at 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 librados: starting objecter 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 Processor -- start 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 -- start start 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 librados: setting wanted keys 2024-03-29T11:36:31.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 1 librados: calling monclient init 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.543+0000 7fa5f4216640 10 monclient: init 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.544+0000 7fa5f4216640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.544+0000 7fa5f4216640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.544+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.544+0000 7fa5f1fb2640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fa5ec0a9d70 0x7fa5ec13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57050/0 (socket says 172.21.15.45:57050) 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f1fb2640 1 -- 172.21.15.45:0/940927171 learned_addr learned my addr 172.21.15.45:0/940927171 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec13f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec1457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec144620 con 0x7fa5ec1457f0 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec141370 con 0x7fa5ec13f650 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec141650 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5e27fc640 10 client.?.objecter ms_handle_connect 0x7fa5ec0a9d70 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5e27fc640 10 client.?.objecter ms_handle_connect 0x7fa5ec1457f0 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:31.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493862430 0 0) 0x7fa5ec144620 con 0x7fa5ec1457f0 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.545+0000 7fa5f4216640 10 monclient(hunting): authenticate will time out at 1047.785077s 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0004eb0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619942802 0 0) 0x7fa5d0004eb0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ec144620 con 0x7fa5ec1457f0 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa5e4001c30 con 0x7fa5ec1457f0 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): dump: 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:31.549 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359524706 0 0) 0x7fa5ec141650 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:36:31.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0004eb0 con 0x7fa5ec0a9d70 2024-03-29T11:36:31.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369291230 0 0) 0x7fa5ec144620 con 0x7fa5ec1457f0 2024-03-29T11:36:31.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 >> v1:172.21.15.45:6790/0 conn(0x7fa5ec0a9d70 legacy=0x7fa5ec13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.546+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 >> v1:172.21.15.190:6789/0 conn(0x7fa5ec13f650 legacy=0x7fa5ec1dedd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:31.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:31.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 1 monclient: found mon.a 2024-03-29T11:36:31.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec13c650 con 0x7fa5ec1457f0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 10 monclient: _renew_subs 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec1404b0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 10 monclient: _renew_subs 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec1452c0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.547+0000 7fa5f4216640 1 librados: init done 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5e4001e00 con 0x7fa5ec1457f0 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fa5e4003bd0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e3fff640 4 set_mon_vals no callback set 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 10 monclient: dump: 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:31.556 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5f4216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.549+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fa5e4019c60 con 0x7fa5ec1457f0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5f4216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fa5e4083be0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5e27fc640 10 client.4389.objecter ms_dispatch 0x7fa5ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5e27fc640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5e27fc640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5e27fc640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5f4216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.550+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d4005080 con 0x7fa5ec1457f0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.551+0000 7fa5f1fb2640 10 monclient: get_auth_request con 0x7fa5d006dd60 auth_method 0 2024-03-29T11:36:31.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.551+0000 7fa5e27fc640 10 client.4389.objecter ms_handle_connect 0x7fa5d006dd60 2024-03-29T11:36:31.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.558+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5e404f2f0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.558+0000 7fa5e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.558+0000 7fa5e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:31.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.883+0000 7fa5f4216640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.883+0000 7fa5f4216640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.883+0000 7fa5f4216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:31.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.883+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5d4005330 con 0x7fa5ec1457f0 2024-03-29T11:36:31.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.884+0000 7fa5e27fc640 1 -- 172.21.15.45:0/940927171 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fa5e40513f0 con 0x7fa5ec1457f0 2024-03-29T11:36:31.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.884+0000 7fa5e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.884+0000 7fa5e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:31.883 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:31.883 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:31.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 10 librados: watch_flush enter 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 10 librados: watch_flush exit 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 10 client.4389.objecter successfully canceled tick 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 >> v1:172.21.15.190:6816/1411598462 conn(0x7fa5d006dd60 legacy=0x7fa5d0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 10 monclient: shutdown 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.886+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 >> v1:172.21.15.45:6789/0 conn(0x7fa5ec1457f0 legacy=0x7fa5ec1e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.887+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 shutdown_connections 2024-03-29T11:36:31.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.887+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 >> 172.21.15.45:0/940927171 conn(0x7fa5ec066a00 msgr2=0x7fa5ec0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.887+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 shutdown_connections 2024-03-29T11:36:31.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.887+0000 7fa5f4216640 1 -- 172.21.15.45:0/940927171 wait complete. 2024-03-29T11:36:31.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:31.887+0000 7fa5f4216640 1 librados: shutdown 2024-03-29T11:36:31.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:38.709 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.874+0000 7f684e216640 10 monclient: get_monmap_and_config 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.874+0000 7f684e216640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.874+0000 7f684e216640 10 monclient: monmap: 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:38.875834+0000 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:38.875834+0000 2024-03-29T11:36:38.875 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.876 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.876 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:38.876 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:38.876 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:38.876 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.876+0000 7f684e216640 1 Processor -- start 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.876+0000 7f684e216640 1 -- start start 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.876+0000 7f684e216640 10 monclient: init 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient(hunting): picked mon.noname-c con 0x7f68480a9930 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient(hunting): picked mon.noname-a con 0x7f6848149dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient(hunting): picked mon.noname-b con 0x7f684814ff40 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68480aae60 con 0x7f6848149dc0 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68480b4310 con 0x7f68480a9930 2024-03-29T11:36:38.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68480afdd0 con 0x7f684814ff40 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f684e216640 10 monclient(hunting): authenticate will time out at 1055.116411s 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f6847fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f684814ff40 0x7f6848152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40030/0 (socket says 172.21.15.45:40030) 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.877+0000 7f6847fff640 1 -- 172.21.15.45:0/961500515 learned_addr learned my addr 172.21.15.45:0/961500515 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560863743 0 0) 0x7f68480b4310 con 0x7f68480a9930 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.878+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6834003520 con 0x7f68480a9930 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544705617 0 0) 0x7f68480afdd0 con 0x7f684814ff40 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68480b4310 con 0x7f684814ff40 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540610219 0 0) 0x7f68480aae60 con 0x7f6848149dc0 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68480afdd0 con 0x7f6848149dc0 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692521337 0 0) 0x7f6834003520 con 0x7f68480a9930 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 >> v1:172.21.15.45:6790/0 conn(0x7f684814ff40 legacy=0x7f6848152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 >> v1:172.21.15.45:6789/0 conn(0x7f6848149dc0 legacy=0x7f684814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684814a950 con 0x7f68480a9930 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f684e216640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f684e216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6838001a10 con 0x7f68480a9930 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: dump: 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.892 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f684e216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6838001c30 con 0x7f68480a9930 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 1 -- 172.21.15.45:0/961500515 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6838001fc0 con 0x7f68480a9930 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f6846ffd640 10 monclient: dump: 2024-03-29T11:36:38.893 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f684e216640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.879+0000 7f684e216640 4 set_mon_vals no callback set 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.880+0000 7f684e216640 10 monclient: shutdown 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.880+0000 7f684e216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.880+0000 7f684e216640 1 -- 172.21.15.45:0/961500515 >> v1:172.21.15.190:6789/0 conn(0x7f68480a9930 legacy=0x7f6848143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.881+0000 7f684e216640 1 -- 172.21.15.45:0/961500515 shutdown_connections 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.881+0000 7f684e216640 1 -- 172.21.15.45:0/961500515 >> 172.21.15.45:0/961500515 conn(0x7f68480b35e0 msgr2=0x7f68480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.881+0000 7f684e216640 1 -- 172.21.15.45:0/961500515 shutdown_connections 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.881+0000 7f684e216640 1 -- 172.21.15.45:0/961500515 wait complete. 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 10 monclient: monmap: 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:38.895 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 librados: starting msgr at 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 librados: starting objecter 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 Processor -- start 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 -- start start 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 librados: setting wanted keys 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 1 librados: calling monclient init 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.882+0000 7f684e216640 10 monclient: init 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.883+0000 7f684e216640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.883+0000 7f684e216640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.883+0000 7f684e216640 10 monclient(hunting): picked mon.noname-a con 0x7f68480a9930 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.883+0000 7f684e216640 10 monclient(hunting): picked mon.noname-b con 0x7f6848149dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 10 monclient(hunting): picked mon.noname-c con 0x7f684814ff40 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f68477fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f68480a9930 0x7f68481dbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:42882/0 (socket says 172.21.15.45:42882) 2024-03-29T11:36:38.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f68477fe640 1 -- 172.21.15.45:0/1442379001 learned_addr learned my addr 172.21.15.45:0/1442379001 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f6847fff640 1 --1- 172.21.15.45:0/1442379001 >> v1:172.21.15.45:6790/0 conn(0x7f684814ff40 0x7f68481e6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40204/0 (socket says 172.21.15.45:40204) 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68481db880 con 0x7f68480a9930 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6848147e90 con 0x7f6848149dc0 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6848148020 con 0x7f684814ff40 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f6844ff9640 10 client.?.objecter ms_handle_connect 0x7f68480a9930 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f6844ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f684e216640 10 monclient(hunting): authenticate will time out at 1055.123850s 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f6844ff9640 10 client.?.objecter ms_handle_connect 0x7f684814ff40 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.884+0000 7f6844ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667083276 0 0) 0x7f68481db880 con 0x7f68480a9930 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6828004eb0 con 0x7f68480a9930 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 client.?.objecter ms_handle_connect 0x7f6848149dc0 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16584621 0 0) 0x7f6828004eb0 con 0x7f68480a9930 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.885+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68481db880 con 0x7f68480a9930 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6838001a50 con 0x7f68480a9930 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.898 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260201685 0 0) 0x7f6848148020 con 0x7f684814ff40 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6828004eb0 con 0x7f684814ff40 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301283129 0 0) 0x7f6848147e90 con 0x7f6848149dc0 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6848148020 con 0x7f6848149dc0 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13422751 0 0) 0x7f6848148020 con 0x7f6848149dc0 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6848147e90 con 0x7f6848149dc0 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6830001870 con 0x7f6848149dc0 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:38.900 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696567664 0 0) 0x7f6828004eb0 con 0x7f684814ff40 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6848148020 con 0x7f684814ff40 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.886+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 848614223 0 0) 0x7f68481db880 con 0x7f68480a9930 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 >> v1:172.21.15.45:6790/0 conn(0x7f684814ff40 legacy=0x7f68481e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 >> v1:172.21.15.190:6789/0 conn(0x7f6848149dc0 legacy=0x7f684813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 1 monclient: found mon.a 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68481dc490 con 0x7f68480a9930 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 10 monclient: _renew_subs 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68481dc680 con 0x7f68480a9930 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 10 monclient: _renew_subs 2024-03-29T11:36:38.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6848140060 con 0x7f68480a9930 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f684e216640 1 librados: init done 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6838003c20 con 0x7f68480a9930 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f6844ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.887+0000 7f68467fc640 4 set_mon_vals no callback set 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.888+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6838003fb0 con 0x7f68480a9930 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.888+0000 7f6844ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.888+0000 7f6844ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.888+0000 7f6844ff9640 10 monclient: dump: 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.903 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.889+0000 7f684e216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.889+0000 7f684e216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.889+0000 7f684e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:38.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.889+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c005080 con 0x7f68480a9930 2024-03-29T11:36:38.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.892+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f6838004110 con 0x7f68480a9930 2024-03-29T11:36:38.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f6838084eb0 con 0x7f68480a9930 2024-03-29T11:36:38.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f6844ff9640 10 client.4395.objecter ms_dispatch 0x7f6848000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f6844ff9640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f683ffff640 10 monclient: get_auth_request con 0x7f682806e1e0 auth_method 0 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f6844ff9640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.893+0000 7f6844ff9640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.894+0000 7f6844ff9640 10 client.4395.objecter ms_handle_connect 0x7f682806e1e0 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.903+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6838050790 con 0x7f68480a9930 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.903+0000 7f6844ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:38.903+0000 7f6844ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:39.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.370+0000 7f684e216640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.370+0000 7f684e216640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.370+0000 7f684e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:39.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.370+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f682c005600 con 0x7f68480a9930 2024-03-29T11:36:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.372+0000 7f6844ff9640 1 -- 172.21.15.45:0/1442379001 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f68380526c0 con 0x7f68480a9930 2024-03-29T11:36:39.371 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.372+0000 7f6844ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.372 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:39.372 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:39.372 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.372+0000 7f6844ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.376+0000 7f684e216640 10 librados: watch_flush enter 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.376+0000 7f684e216640 10 librados: watch_flush exit 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.376+0000 7f684e216640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 10 client.4395.objecter successfully canceled tick 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 >> v1:172.21.15.190:6816/1411598462 conn(0x7f682806e1e0 legacy=0x7f68280706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 10 monclient: shutdown 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 >> v1:172.21.15.45:6789/0 conn(0x7f68480a9930 legacy=0x7f68481dbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 shutdown_connections 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 >> 172.21.15.45:0/1442379001 conn(0x7f68480b35e0 msgr2=0x7f68480adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 shutdown_connections 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 -- 172.21.15.45:0/1442379001 wait complete. 2024-03-29T11:36:39.377 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:39.377+0000 7f684e216640 1 librados: shutdown 2024-03-29T11:36:39.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:46.200 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.344+0000 7f0c7db7c640 10 monclient: get_monmap_and_config 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.344+0000 7f0c7db7c640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.344+0000 7f0c7db7c640 10 monclient: monmap: 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:46.345706+0000 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:46.345706+0000 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:46.344 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:46.345 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:46.345 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.345+0000 7f0c7db7c640 1 Processor -- start 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.345+0000 7f0c7db7c640 1 -- start start 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.345+0000 7f0c7db7c640 10 monclient: init 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.346+0000 7f0c7db7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.346+0000 7f0c7db7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:46.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.346+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c780b4790 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.346+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c780aea90 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.346+0000 7f0c76ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0c780aea90 0x7f0c780af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45854/0 (socket says 172.21.15.45:45854) 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c777fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0c780b4790 0x7f0c780b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37050/0 (socket says 172.21.15.45:37050) 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c780b39f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780b0c60 con 0x7f0c780b39f0 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c76ffd640 1 -- 172.21.15.45:0/4221925112 learned_addr learned my addr 172.21.15.45:0/4221925112 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780af020 con 0x7f0c780aea90 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7813fbd0 con 0x7f0c780b4790 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c7db7c640 10 monclient(hunting): authenticate will time out at 1062.586441s 2024-03-29T11:36:46.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526203870 0 0) 0x7f0c7813fbd0 con 0x7f0c780b4790 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60003520 con 0x7f0c780b4790 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074203077 0 0) 0x7f0c780af020 con 0x7f0c780aea90 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:36:46.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.347+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7813fbd0 con 0x7f0c780aea90 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826643367 0 0) 0x7f0c780b0c60 con 0x7f0c780b39f0 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c780af020 con 0x7f0c780b39f0 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4149122022 0 0) 0x7f0c60003520 con 0x7f0c780b4790 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 >> v1:172.21.15.190:6789/0 conn(0x7f0c780aea90 legacy=0x7f0c780af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 >> v1:172.21.15.45:6789/0 conn(0x7f0c780b39f0 legacy=0x7f0c7813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:36:46.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7813fd60 con 0x7f0c780b4790 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0c68001540 con 0x7f0c780b4790 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c767fc640 10 monclient: dump: 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.352 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c7db7c640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.348+0000 7f0c7db7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c680016a0 con 0x7f0c780b4790 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 1 -- 172.21.15.45:0/4221925112 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0c68001af0 con 0x7f0c780b4790 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c767fc640 10 monclient: dump: 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:46.353 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c7db7c640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.349+0000 7f0c7db7c640 4 set_mon_vals no callback set 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 10 monclient: shutdown 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 >> v1:172.21.15.45:6790/0 conn(0x7f0c780b4790 legacy=0x7f0c780b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 shutdown_connections 2024-03-29T11:36:46.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 >> 172.21.15.45:0/4221925112 conn(0x7f0c78066a00 msgr2=0x7f0c78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:46.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 shutdown_connections 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.350+0000 7f0c7db7c640 1 -- 172.21.15.45:0/4221925112 wait complete. 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.351+0000 7f0c7db7c640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.352+0000 7f0c7db7c640 10 monclient: monmap: 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:46.360 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.352+0000 7f0c7db7c640 1 librados: starting msgr at 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.352+0000 7f0c7db7c640 1 librados: starting objecter 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.352+0000 7f0c7db7c640 1 Processor -- start 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.353+0000 7f0c7db7c640 1 -- start start 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.353+0000 7f0c7db7c640 1 librados: setting wanted keys 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.353+0000 7f0c7db7c640 1 librados: calling monclient init 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.353+0000 7f0c7db7c640 10 monclient: init 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.354+0000 7f0c7db7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.354+0000 7f0c7db7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.354+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c780b39f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:46.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.354+0000 7f0c777fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0c780b39f0 0x7f0c781d7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:46766/0 (socket says 172.21.15.45:46766) 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.354+0000 7f0c777fe640 1 -- 172.21.15.45:0/3106070078 learned_addr learned my addr 172.21.15.45:0/3106070078 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 client.?.objecter ms_handle_connect 0x7f0c780b39f0 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c780b4790 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c781d3dd0 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c78137030 con 0x7f0c780b4790 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 client.?.objecter ms_handle_connect 0x7f0c780b4790 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c781d74e0 con 0x7f0c780b39f0 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780ad670 con 0x7f0c781d3dd0 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c7db7c640 10 monclient(hunting): authenticate will time out at 1062.594735s 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772764721 0 0) 0x7f0c78137030 con 0x7f0c780b4790 2024-03-29T11:36:46.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.355+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c64004eb0 con 0x7f0c780b4790 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660393964 0 0) 0x7f0c781d74e0 con 0x7f0c780b39f0 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c78137030 con 0x7f0c780b39f0 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.356+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2461109756 0 0) 0x7f0c64004eb0 con 0x7f0c780b4790 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c781d74e0 con 0x7f0c780b4790 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 10 client.?.objecter ms_handle_connect 0x7f0c781d3dd0 2024-03-29T11:36:46.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0c6c001970 con 0x7f0c780b4790 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.357+0000 7f0c5ffff640 10 monclient(hunting): dump: 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:46.364 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4058232215 0 0) 0x7f0c78137030 con 0x7f0c780b39f0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c64004eb0 con 0x7f0c780b39f0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1601112383 0 0) 0x7f0c781d74e0 con 0x7f0c780b4790 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 >> v1:172.21.15.45:6790/0 conn(0x7f0c781d3dd0 legacy=0x7f0c781e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 >> v1:172.21.15.190:6789/0 conn(0x7f0c780b39f0 legacy=0x7f0c781d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 monclient: found mon.a 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c781d8110 con 0x7f0c780b4790 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.365 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 10 monclient: _renew_subs 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781d4e10 con 0x7f0c780b4790 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 10 monclient: _renew_subs 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c780ae310 con 0x7f0c780b4790 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.358+0000 7f0c7db7c640 1 librados: init done 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c6c001ad0 con 0x7f0c780b4790 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0c6c004a90 con 0x7f0c780b4790 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.366 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c5ffff640 10 monclient: dump: 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c7db7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.359+0000 7f0c75ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:46.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.360+0000 7f0c7db7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.360+0000 7f0c7db7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:46.368 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.360+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c781d7340 con 0x7f0c780b4790 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.363+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f0c6c01aa60 con 0x7f0c780b4790 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f0c6c084830 con 0x7f0c780b4790 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c5ffff640 10 client.4401.objecter ms_dispatch 0x7f0c78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c5ffff640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c5ffff640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c5ffff640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.364+0000 7f0c777fe640 10 monclient: get_auth_request con 0x7f0c6406dd40 auth_method 0 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.365+0000 7f0c5ffff640 10 client.4401.objecter ms_handle_connect 0x7f0c6406dd40 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.369+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c6c050110 con 0x7f0c780b4790 2024-03-29T11:36:46.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.369+0000 7f0c5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.370 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.369+0000 7f0c5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:46.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.688+0000 7f0c7db7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.688+0000 7f0c7db7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.688+0000 7f0c7db7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:36:46.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.688+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c781d7340 con 0x7f0c780b4790 2024-03-29T11:36:46.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.689+0000 7f0c5ffff640 1 -- 172.21.15.45:0/3106070078 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f0c781d7340 con 0x7f0c780b4790 2024-03-29T11:36:46.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.689+0000 7f0c5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.689+0000 7f0c5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:46.689 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:46.690 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.692+0000 7f0c7db7c640 10 librados: watch_flush enter 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 10 librados: watch_flush exit 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 >> v1:172.21.15.190:6816/1411598462 conn(0x7f0c6406dd40 legacy=0x7f0c64070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 10 monclient: shutdown 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:46.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 >> v1:172.21.15.45:6789/0 conn(0x7f0c780b4790 legacy=0x7f0c781d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 shutdown_connections 2024-03-29T11:36:46.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 >> 172.21.15.45:0/3106070078 conn(0x7f0c78066a00 msgr2=0x7f0c780b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:46.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 shutdown_connections 2024-03-29T11:36:46.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 -- 172.21.15.45:0/3106070078 wait complete. 2024-03-29T11:36:46.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:46.693+0000 7f0c7db7c640 1 librados: shutdown 2024-03-29T11:36:46.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:53.514 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:53.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 10 monclient: get_monmap_and_config 2024-03-29T11:36:53.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 10 monclient: build_initial_monmap 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 10 monclient: monmap: 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:36:53.649224+0000 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:36:53.649224+0000 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:36:53.648 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 1 Processor -- start 2024-03-29T11:36:53.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 1 -- start start 2024-03-29T11:36:53.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.648+0000 7fe6f0d82640 10 monclient: init 2024-03-29T11:36:53.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6f0d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:53.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6f0d82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ec0a9750 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6ea59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6ec0a9750 0x7fe6ec13dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45924/0 (socket says 172.21.15.45:45924) 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ec13e1e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.649+0000 7fe6ea59c640 1 -- 172.21.15.45:0/2592987337 learned_addr learned my addr 172.21.15.45:0/2592987337 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ec071c30 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec0ab300 con 0x7fe6ec13e1e0 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec0ac2f0 con 0x7fe6ec071c30 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec0ad8d0 con 0x7fe6ec0a9750 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:53.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.650+0000 7fe6f0d82640 10 monclient(hunting): authenticate will time out at 1069.889602s 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409098182 0 0) 0x7fe6ec0ad8d0 con 0x7fe6ec0a9750 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8003520 con 0x7fe6ec0a9750 2024-03-29T11:36:53.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003918295 0 0) 0x7fe6d8003520 con 0x7fe6ec0a9750 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 >> v1:172.21.15.190:6789/0 conn(0x7fe6ec071c30 legacy=0x7fe6ec0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.651+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 >> v1:172.21.15.45:6789/0 conn(0x7fe6ec13e1e0 legacy=0x7fe6ec0ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 1 monclient: found mon.noname-b 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec0ada60 con 0x7fe6ec0a9750 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe6dc001670 con 0x7fe6ec0a9750 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:53.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.652+0000 7fe6e9d9b640 10 monclient: dump: 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6f0d82640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6f0d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6dc001900 con 0x7fe6ec0a9750 2024-03-29T11:36:53.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 1 -- 172.21.15.45:0/2592987337 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe6dc001cd0 con 0x7fe6ec0a9750 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6e9d9b640 10 monclient: dump: 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6f0d82640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.653+0000 7fe6f0d82640 4 set_mon_vals no callback set 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 10 monclient: shutdown 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 >> v1:172.21.15.45:6790/0 conn(0x7fe6ec0a9750 legacy=0x7fe6ec13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 shutdown_connections 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 >> 172.21.15.45:0/2592987337 conn(0x7fe6ec06b660 msgr2=0x7fe6ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 shutdown_connections 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.654+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2592987337 wait complete. 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.655+0000 7fe6f0d82640 10 monclient: build_initial_monmap 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.655+0000 7fe6f0d82640 10 monclient: monmap: 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:36:53.668 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 librados: starting msgr at 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 librados: starting objecter 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 Processor -- start 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 -- start start 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 librados: setting wanted keys 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 1 librados: calling monclient init 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.656+0000 7fe6f0d82640 10 monclient: init 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:53.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ec071c30 addr v1:172.21.15.45:6790/0 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6ea59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe6ec071c30 0x7fe6ec06f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46904/0 (socket says 172.21.15.45:46904) 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ec13e1e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ec141780 addr v1:172.21.15.190:6789/0 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.657+0000 7fe6f0d82640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec070ff0 con 0x7fe6ec13e1e0 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6f0d82640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec071260 con 0x7fe6ec141780 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6ead9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe6ec141780 0x7fe6ec1e7360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:37386/0 (socket says 172.21.15.45:37386) 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6ead9d640 1 -- 172.21.15.45:0/2002075270 learned_addr learned my addr 172.21.15.45:0/2002075270 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec11f080 con 0x7fe6ec071c30 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6f0d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6f0d82640 10 monclient(hunting): authenticate will time out at 1069.897498s 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 client.?.objecter ms_handle_connect 0x7fe6ec141780 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:53.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 client.?.objecter ms_handle_connect 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608275598 0 0) 0x7fe6ec11f080 con 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.658+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6cc004eb0 con 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142012830 0 0) 0x7fe6cc004eb0 con 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ec11f080 con 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe6dc001630 con 0x7fe6ec071c30 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:53.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): dump: 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798612792 0 0) 0x7fe6ec071260 con 0x7fe6ec141780 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:53.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.659+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6cc004eb0 con 0x7fe6ec141780 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2688977127 0 0) 0x7fe6ec11f080 con 0x7fe6ec071c30 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 >> v1:172.21.15.190:6789/0 conn(0x7fe6ec141780 legacy=0x7fe6ec1e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 >> v1:172.21.15.45:6789/0 conn(0x7fe6ec13e1e0 legacy=0x7fe6ec11e1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 monclient: found mon.c 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec06ffb0 con 0x7fe6ec071c30 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:53.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6f0d82640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6f0d82640 10 monclient: _renew_subs 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6f0d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ec06ec00 con 0x7fe6ec071c30 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6dc001790 con 0x7fe6ec071c30 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe6dc004e50 con 0x7fe6ec071c30 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.660+0000 7fe6e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6e37fe640 10 monclient: dump: 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.674 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6f0d82640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6f0d82640 10 monclient: _renew_subs 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6f0d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6f0d82640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ec071860 con 0x7fe6ec071c30 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6f0d82640 1 librados: init done 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.661+0000 7fe6e959a640 4 set_mon_vals no callback set 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.663+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe6dc01ae20 con 0x7fe6ec071c30 2024-03-29T11:36:53.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.663+0000 7fe6e0d99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.663+0000 7fe6e0d99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.663+0000 7fe6e0d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.663+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c4005080 con 0x7fe6ec071c30 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe6dc084bb0 con 0x7fe6ec071c30 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 10 client.4421.objecter ms_dispatch 0x7fe6ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e1d9b640 10 monclient: get_auth_request con 0x7fe6cc06dd00 auth_method 0 2024-03-29T11:36:53.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.664+0000 7fe6e37fe640 10 client.4421.objecter ms_handle_connect 0x7fe6cc06dd00 2024-03-29T11:36:53.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.670+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6dc050490 con 0x7fe6ec071c30 2024-03-29T11:36:53.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.670+0000 7fe6e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.670+0000 7fe6e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:53.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.975+0000 7fe6e0d99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.975+0000 7fe6e0d99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.975+0000 7fe6e0d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:36:53.975 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.975+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6c4005330 con 0x7fe6ec071c30 2024-03-29T11:36:53.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.976+0000 7fe6e37fe640 1 -- 172.21.15.45:0/2002075270 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe6dc0523c0 con 0x7fe6ec071c30 2024-03-29T11:36:53.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.976+0000 7fe6e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.976+0000 7fe6e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:53.976 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:36:53.976 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 10 librados: watch_flush enter 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 10 librados: watch_flush exit 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe6cc06dd00 legacy=0x7fe6cc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 10 monclient: shutdown 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 >> v1:172.21.15.45:6790/0 conn(0x7fe6ec071c30 legacy=0x7fe6ec06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 shutdown_connections 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 >> 172.21.15.45:0/2002075270 conn(0x7fe6ec06b660 msgr2=0x7fe6ec06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 shutdown_connections 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 -- 172.21.15.45:0/2002075270 wait complete. 2024-03-29T11:36:53.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:36:53.979+0000 7fe6e0d99640 1 librados: shutdown 2024-03-29T11:36:53.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:00.800 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.950+0000 7f0ec1a4a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.950+0000 7f0ec1a4a640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.950+0000 7f0ec1a4a640 10 monclient: monmap: 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:00.951508+0000 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:00.951508+0000 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:00.950 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:00.951 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:00.951 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.951+0000 7f0ec1a4a640 1 Processor -- start 2024-03-29T11:37:00.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.952+0000 7f0ec1a4a640 1 -- start start 2024-03-29T11:37:00.951 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.952+0000 7f0ec1a4a640 10 monclient: init 2024-03-29T11:37:00.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f0ebc1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f0ebc13f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f0ebc0a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0aaef0 con 0x7f0ebc13f650 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc145bd0 con 0x7f0ebc0a9d70 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc145eb0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.953+0000 7f0ec1a4a640 10 monclient(hunting): authenticate will time out at 1077.192605s 2024-03-29T11:37:00.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.955+0000 7f0ebaffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0ebc1457f0 0x7f0ebc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33248/0 (socket says 172.21.15.45:33248) 2024-03-29T11:37:00.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.955+0000 7f0ebaffd640 1 -- 172.21.15.45:0/1184198956 learned_addr learned my addr 172.21.15.45:0/1184198956 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:00.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278989725 0 0) 0x7f0ebc145eb0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:37:00.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.956+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4003520 con 0x7f0ebc1457f0 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146990014 0 0) 0x7f0ea4003520 con 0x7f0ebc1457f0 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 >> v1:172.21.15.190:6789/0 conn(0x7f0ebc0a9d70 legacy=0x7f0ebc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 >> v1:172.21.15.45:6789/0 conn(0x7f0ebc13f650 legacy=0x7f0ebc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:00.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc146340 con 0x7f0ebc1457f0 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.958+0000 7f0eb9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0eb4001690 con 0x7f0ebc1457f0 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0ec1a4a640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0ec1a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0eb9ffb640 10 monclient: dump: 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.960 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.959+0000 7f0ec1a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:00.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eb4001920 con 0x7f0ebc1457f0 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 1 -- 172.21.15.45:0/1184198956 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0eb4001cb0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0eb9ffb640 10 monclient: dump: 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:00.963 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0ec1a4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.960+0000 7f0ec1a4a640 4 set_mon_vals no callback set 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 10 monclient: shutdown 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/1184198956 >> v1:172.21.15.45:6790/0 conn(0x7f0ebc1457f0 legacy=0x7f0ebc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/1184198956 shutdown_connections 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/1184198956 >> 172.21.15.45:0/1184198956 conn(0x7f0ebc066a00 msgr2=0x7f0ebc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/1184198956 shutdown_connections 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.961+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/1184198956 wait complete. 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.962+0000 7f0ec1a4a640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.962+0000 7f0ec1a4a640 10 monclient: monmap: 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:00.964 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 librados: starting msgr at 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 librados: starting objecter 2024-03-29T11:37:00.965 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 Processor -- start 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 -- start start 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 librados: setting wanted keys 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 1 librados: calling monclient init 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.963+0000 7f0ec1a4a640 10 monclient: init 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f0ebc0a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f0ebc13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f0ebc1457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc13b1e0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0718f0 con 0x7f0ebc0a9d70 2024-03-29T11:37:00.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc071bd0 con 0x7f0ebc13f650 2024-03-29T11:37:00.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.964+0000 7f0ec1a4a640 10 monclient(hunting): authenticate will time out at 1077.203766s 2024-03-29T11:37:00.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.967+0000 7f0eba7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f0ebc13f650 0x7f0ebc13a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33716/0 (socket says 172.21.15.45:33716) 2024-03-29T11:37:00.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.967+0000 7f0eba7fc640 1 -- 172.21.15.45:0/2534441595 learned_addr learned my addr 172.21.15.45:0/2534441595 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:00.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 client.?.objecter ms_handle_connect 0x7f0ebc13f650 2024-03-29T11:37:00.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 client.?.objecter ms_handle_connect 0x7f0ebc1457f0 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759722828 0 0) 0x7f0ebc071bd0 con 0x7f0ebc13f650 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea8004eb0 con 0x7f0ebc13f650 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861769647 0 0) 0x7f0ebc13b1e0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.968+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ebc071bd0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552364276 0 0) 0x7f0ebc071bd0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc13b1e0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0eac001890 con 0x7f0ebc1457f0 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): dump: 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:00.971 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276704779 0 0) 0x7f0ea8004eb0 con 0x7f0ebc13f650 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ebc071bd0 con 0x7f0ebc13f650 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0eb0004080 con 0x7f0ebc13f650 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.969+0000 7f0ea3fff640 10 monclient(hunting): dump: 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.972 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.973 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:00.973 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:00.973 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:00.973 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3225393114 0 0) 0x7f0ebc13b1e0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 >> v1:172.21.15.45:6790/0 conn(0x7f0ebc13f650 legacy=0x7f0ebc13a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 >> v1:172.21.15.190:6789/0 conn(0x7f0ebc0a9d70 legacy=0x7f0ebc070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 1 monclient: found mon.a 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.976 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc06ff20 con 0x7f0ebc1457f0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 10 monclient: _renew_subs 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc1dea50 con 0x7f0ebc1457f0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 10 monclient: _renew_subs 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc070d50 con 0x7f0ebc1457f0 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.971+0000 7f0ec1a4a640 1 librados: init done 2024-03-29T11:37:00.977 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eac001da0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0eac003c10 con 0x7f0ebc1457f0 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0ea3fff640 10 monclient: dump: 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:00.978 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.972+0000 7f0eb8ff9640 4 set_mon_vals no callback set 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.973+0000 7f0ec1a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.974+0000 7f0ec1a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.974+0000 7f0ec1a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.974+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c005080 con 0x7f0ebc1457f0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.974+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f0eac019be0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f0eac083b10 con 0x7f0ebc1457f0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ea3fff640 10 client.4404.objecter ms_dispatch 0x7f0ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ea3fff640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ea3fff640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ea3fff640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.975+0000 7f0ebaffd640 10 monclient: get_auth_request con 0x7f0ea806ded0 auth_method 0 2024-03-29T11:37:00.979 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.976+0000 7f0ea3fff640 10 client.4404.objecter ms_handle_connect 0x7f0ea806ded0 2024-03-29T11:37:00.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.986+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0eac04f3f0 con 0x7f0ebc1457f0 2024-03-29T11:37:00.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.986+0000 7f0ea3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:00.986+0000 7f0ea3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:01.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.314+0000 7f0ec1a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.315+0000 7f0ec1a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.315+0000 7f0ec1a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:01.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.315+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e9c005330 con 0x7f0ebc1457f0 2024-03-29T11:37:01.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.316+0000 7f0ea3fff640 1 -- 172.21.15.45:0/2534441595 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f0eac051320 con 0x7f0ebc1457f0 2024-03-29T11:37:01.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.316+0000 7f0ea3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.316+0000 7f0ea3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:01.316 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:01.316 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:01.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.319+0000 7f0ec1a4a640 10 librados: watch_flush enter 2024-03-29T11:37:01.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.319+0000 7f0ec1a4a640 10 librados: watch_flush exit 2024-03-29T11:37:01.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.319+0000 7f0ec1a4a640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:37:01.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.319+0000 7f0ec1a4a640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.319+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 >> v1:172.21.15.190:6816/1411598462 conn(0x7f0ea806ded0 legacy=0x7f0ea8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 10 monclient: shutdown 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 >> v1:172.21.15.45:6789/0 conn(0x7f0ebc1457f0 legacy=0x7f0ebc13aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 shutdown_connections 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 >> 172.21.15.45:0/2534441595 conn(0x7f0ebc066a00 msgr2=0x7f0ebc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 shutdown_connections 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 -- 172.21.15.45:0/2534441595 wait complete. 2024-03-29T11:37:01.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:01.320+0000 7f0ec1a4a640 1 librados: shutdown 2024-03-29T11:37:01.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:08.144 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:08.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.289+0000 7f277d34d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.289+0000 7f277d34d640 10 monclient: build_initial_monmap 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.289+0000 7f277d34d640 10 monclient: monmap: 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:08.291148+0000 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:08.291148+0000 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:08.290 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.291+0000 7f277d34d640 1 Processor -- start 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.291+0000 7f277d34d640 1 -- start start 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.291+0000 7f277d34d640 10 monclient: init 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:08.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-c con 0x7f277806f050 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-b con 0x7f2778070240 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-a con 0x7f277806f430 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277813f400 con 0x7f277806f430 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27781429c0 con 0x7f277806f050 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277806ff80 con 0x7f2778070240 2024-03-29T11:37:08.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:08.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.292+0000 7f277d34d640 10 monclient(hunting): authenticate will time out at 1084.531200s 2024-03-29T11:37:08.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.294+0000 7f27767fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2778070240 0x7f27780ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58348/0 (socket says 172.21.15.45:58348) 2024-03-29T11:37:08.294 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.294+0000 7f27767fc640 1 -- 172.21.15.45:0/627270735 learned_addr learned my addr 172.21.15.45:0/627270735 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:08.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811589301 0 0) 0x7f27781429c0 con 0x7f277806f050 2024-03-29T11:37:08.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:37:08.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2760003520 con 0x7f277806f050 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1748245447 0 0) 0x7f2760003520 con 0x7f277806f050 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 >> v1:172.21.15.45:6790/0 conn(0x7f2778070240 legacy=0x7f27780ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.295+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 >> v1:172.21.15.45:6789/0 conn(0x7f277806f430 legacy=0x7f277814a630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 1 monclient: found mon.noname-c 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:37:08.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780aef70 con 0x7f277806f050 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2770001a10 con 0x7f277806f050 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f2775ffb640 10 monclient: dump: 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.299 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f277d34d640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.296+0000 7f277d34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2770001ca0 con 0x7f277806f050 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f277d34d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f277d34d640 4 set_mon_vals no callback set 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 1 -- 172.21.15.45:0/627270735 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2770002030 con 0x7f277806f050 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.297+0000 7f2775ffb640 10 monclient: dump: 2024-03-29T11:37:08.300 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.298+0000 7f277d34d640 10 monclient: shutdown 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.298+0000 7f277d34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:08.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.298+0000 7f277d34d640 1 -- 172.21.15.45:0/627270735 >> v1:172.21.15.190:6789/0 conn(0x7f277806f050 legacy=0x7f27781422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.300+0000 7f277d34d640 1 -- 172.21.15.45:0/627270735 shutdown_connections 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.300+0000 7f277d34d640 1 -- 172.21.15.45:0/627270735 >> 172.21.15.45:0/627270735 conn(0x7f27780b4930 msgr2=0x7f27780b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.300+0000 7f277d34d640 1 -- 172.21.15.45:0/627270735 shutdown_connections 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.300+0000 7f277d34d640 1 -- 172.21.15.45:0/627270735 wait complete. 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.301+0000 7f277d34d640 10 monclient: build_initial_monmap 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.301+0000 7f277d34d640 10 monclient: monmap: 2024-03-29T11:37:08.303 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.301+0000 7f277d34d640 1 librados: starting msgr at 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.301+0000 7f277d34d640 1 librados: starting objecter 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.302+0000 7f277d34d640 1 Processor -- start 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.302+0000 7f277d34d640 1 -- start start 2024-03-29T11:37:08.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.302+0000 7f277d34d640 1 librados: setting wanted keys 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.302+0000 7f277d34d640 1 librados: calling monclient init 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.302+0000 7f277d34d640 10 monclient: init 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-c con 0x7f277806f050 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f2776ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f277806f050 0x7f2778142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37282/0 (socket says 172.21.15.45:37282) 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-a con 0x7f277806f430 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 10 monclient(hunting): picked mon.noname-b con 0x7f2778070240 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f2776ffd640 1 -- 172.21.15.45:0/1146655453 learned_addr learned my addr 172.21.15.45:0/1146655453 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f275f7fe640 10 client.?.objecter ms_handle_connect 0x7f277806f050 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f275f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27781e3b30 con 0x7f277806f430 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27781e49b0 con 0x7f2778070240 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.303+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27781e3d80 con 0x7f277806f050 2024-03-29T11:37:08.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.304+0000 7f277d34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.304+0000 7f277d34d640 10 monclient(hunting): authenticate will time out at 1084.543175s 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481121663 0 0) 0x7f27781e3d80 con 0x7f277806f050 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.305+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2764004eb0 con 0x7f277806f050 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275104819 0 0) 0x7f2764004eb0 con 0x7f277806f050 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27781e3d80 con 0x7f277806f050 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2770001a10 con 0x7f277806f050 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:08.307 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:08.308 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:08.308 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:08.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1222999952 0 0) 0x7f27781e3d80 con 0x7f277806f050 2024-03-29T11:37:08.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.306+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 >> v1:172.21.15.190:6789/0 conn(0x7f2778070240 legacy=0x7f27781e0ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 >> v1:172.21.15.45:6789/0 conn(0x7f277806f430 legacy=0x7f27781e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 1 monclient: found mon.c 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780ad0b0 con 0x7f277806f050 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f275f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.308+0000 7f277d34d640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:37:08.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 10 monclient: _renew_subs 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27780aa790 con 0x7f277806f050 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 10 monclient: _renew_subs 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27780aaae0 con 0x7f277806f050 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f277d34d640 1 librados: init done 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f275f7fe640 10 client.4430.objecter ms_handle_connect 0x7f277806f430 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f275f7fe640 10 client.4430.objecter resend_mon_ops 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2770003ce0 con 0x7f277806f050 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f275f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.309+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2770004130 con 0x7f277806f050 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.310+0000 7f275f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.310+0000 7f275f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:08.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.310+0000 7f27757fa640 4 set_mon_vals no callback set 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.311+0000 7f275f7fe640 10 monclient: dump: 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:08.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.311+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f2770004290 con 0x7f277806f050 2024-03-29T11:37:08.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2770085020 con 0x7f277806f050 2024-03-29T11:37:08.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f27767fc640 10 monclient: get_auth_request con 0x7f276406dbe0 auth_method 0 2024-03-29T11:37:08.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f275f7fe640 10 client.4430.objecter ms_dispatch 0x7f2778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f275f7fe640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f275f7fe640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f275f7fe640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.312+0000 7f277d34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.313+0000 7f277d34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.313+0000 7f277d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.313+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2758005080 con 0x7f277806f050 2024-03-29T11:37:08.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.313+0000 7f275f7fe640 10 client.4430.objecter ms_handle_connect 0x7f276406dbe0 2024-03-29T11:37:08.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.322+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2770050900 con 0x7f277806f050 2024-03-29T11:37:08.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.322+0000 7f275f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.322+0000 7f275f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:08.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.655+0000 7f277d34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.655+0000 7f277d34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.655+0000 7f277d34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:08.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.655+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2758005330 con 0x7f277806f050 2024-03-29T11:37:08.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.656+0000 7f275f7fe640 1 -- 172.21.15.45:0/1146655453 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f2770052830 con 0x7f277806f050 2024-03-29T11:37:08.655 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.656+0000 7f275f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.656+0000 7f275f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:08.656 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:08.656 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:08.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.660+0000 7f277d34d640 10 librados: watch_flush enter 2024-03-29T11:37:08.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.661+0000 7f277d34d640 10 librados: watch_flush exit 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.661+0000 7f277d34d640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.661+0000 7f277d34d640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 >> v1:172.21.15.190:6816/1411598462 conn(0x7f276406dbe0 legacy=0x7f27640700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 10 monclient: shutdown 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 >> v1:172.21.15.45:6790/0 conn(0x7f277806f050 legacy=0x7f2778142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 shutdown_connections 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 >> 172.21.15.45:0/1146655453 conn(0x7f27780b4930 msgr2=0x7f2778072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 shutdown_connections 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 -- 172.21.15.45:0/1146655453 wait complete. 2024-03-29T11:37:08.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:08.662+0000 7f277d34d640 1 librados: shutdown 2024-03-29T11:37:08.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:15.492 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.637+0000 7fb5b3e55640 10 monclient: get_monmap_and_config 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.637+0000 7fb5b3e55640 10 monclient: build_initial_monmap 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.637+0000 7fb5b3e55640 10 monclient: monmap: 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:15.637531+0000 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:15.637531+0000 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.638 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.639+0000 7fb5b3e55640 1 Processor -- start 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.641+0000 7fb5b3e55640 1 -- start start 2024-03-29T11:37:15.639 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.641+0000 7fb5b3e55640 10 monclient: init 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac0ae250 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac13f970 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ac142800 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0afe00 con 0x7fb5ac0ae250 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac142c10 con 0x7fb5ac142800 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0afb20 con 0x7fb5ac13f970 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:15.641 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.642+0000 7fb5b3e55640 10 monclient(hunting): authenticate will time out at 1091.880300s 2024-03-29T11:37:15.645 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.646+0000 7fb5b1bf1640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb5ac0ae250 0x7fb5ac13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:35627/0 (socket says 172.21.15.45:35627) 2024-03-29T11:37:15.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.646+0000 7fb5b1bf1640 1 -- 172.21.15.45:0/3313009151 learned_addr learned my addr 172.21.15.45:0/3313009151 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:15.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019209631 0 0) 0x7fb5ac0afe00 con 0x7fb5ac0ae250 2024-03-29T11:37:15.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.646 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:37:15.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598003520 con 0x7fb5ac0ae250 2024-03-29T11:37:15.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313097863 0 0) 0x7fb5ac0afb20 con 0x7fb5ac13f970 2024-03-29T11:37:15.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ac0afe00 con 0x7fb5ac13f970 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820085302 0 0) 0x7fb598003520 con 0x7fb5ac0ae250 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 >> v1:172.21.15.45:6790/0 conn(0x7fb5ac13f970 legacy=0x7fb5ac1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 >> v1:172.21.15.190:6789/0 conn(0x7fb5ac142800 legacy=0x7fb5ac14da10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: _finish_hunting 0 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 monclient: found mon.noname-a 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0aab80 con 0x7fb5ac0ae250 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: _finish_auth 0 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: _check_auth_tickets 2024-03-29T11:37:15.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb5a8001690 con 0x7fb5ac0ae250 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.648+0000 7fb5b0bef640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 10 monclient: dump: 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:15.649 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5a80018b0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 1 -- 172.21.15.45:0/3313009151 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb5a8001c40 con 0x7fb5ac0ae250 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.649+0000 7fb5b0bef640 10 monclient: dump: 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:15.650 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:15.651 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:15.651 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.651+0000 7fb5b3e55640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.651+0000 7fb5b3e55640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.651+0000 7fb5b3e55640 4 set_mon_vals no callback set 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.652+0000 7fb5b3e55640 10 monclient: shutdown 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.652+0000 7fb5b3e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.652+0000 7fb5b3e55640 1 -- 172.21.15.45:0/3313009151 >> v1:172.21.15.45:6789/0 conn(0x7fb5ac0ae250 legacy=0x7fb5ac13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.652+0000 7fb5b3e55640 1 -- 172.21.15.45:0/3313009151 shutdown_connections 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.653+0000 7fb5b3e55640 1 -- 172.21.15.45:0/3313009151 >> 172.21.15.45:0/3313009151 conn(0x7fb5ac0b35e0 msgr2=0x7fb5ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:15.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.653+0000 7fb5b3e55640 1 -- 172.21.15.45:0/3313009151 shutdown_connections 2024-03-29T11:37:15.654 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.653+0000 7fb5b3e55640 1 -- 172.21.15.45:0/3313009151 wait complete. 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.656+0000 7fb5b3e55640 10 monclient: build_initial_monmap 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.656+0000 7fb5b3e55640 10 monclient: monmap: 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 librados: starting msgr at 2024-03-29T11:37:15.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 librados: starting objecter 2024-03-29T11:37:15.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 Processor -- start 2024-03-29T11:37:15.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 -- start start 2024-03-29T11:37:15.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 librados: setting wanted keys 2024-03-29T11:37:15.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 1 librados: calling monclient init 2024-03-29T11:37:15.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.657+0000 7fb5b3e55640 10 monclient: init 2024-03-29T11:37:15.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.658+0000 7fb5b3e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:15.658 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.658+0000 7fb5b3e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:15.659 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.660+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac0ae250 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:15.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac13f970 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:15.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ac142800 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac13b060 con 0x7fb5ac13f970 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac1dc3f0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac13bef0 con 0x7fb5ac142800 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b3e55640 10 monclient(hunting): authenticate will time out at 1091.902232s 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b1bf1640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb5ac0ae250 0x7fb5ac144720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:53489/0 (socket says 172.21.15.45:53489) 2024-03-29T11:37:15.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.664+0000 7fb5b1bf1640 1 -- 172.21.15.45:0/950941317 learned_addr learned my addr 172.21.15.45:0/950941317 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.666+0000 7fb5a27fc640 10 client.?.objecter ms_handle_connect 0x7fb5ac0ae250 2024-03-29T11:37:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:15.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940141479 0 0) 0x7fb5ac1dc3f0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:37:15.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb590004eb0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 client.?.objecter ms_handle_connect 0x7fb5ac13f970 2024-03-29T11:37:15.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950977111 0 0) 0x7fb590004eb0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5ac1dc3f0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb5a80016d0 con 0x7fb5ac0ae250 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): dump: 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.668 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405314087 0 0) 0x7fb5ac13b060 con 0x7fb5ac13f970 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.667+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb590004eb0 con 0x7fb5ac13f970 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058541095 0 0) 0x7fb590004eb0 con 0x7fb5ac13f970 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5ac13b060 con 0x7fb5ac13f970 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb59c0017e0 con 0x7fb5ac13f970 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): dump: 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 client.?.objecter ms_handle_connect 0x7fb5ac142800 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:15.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133076923 0 0) 0x7fb5ac13bef0 con 0x7fb5ac142800 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb590004eb0 con 0x7fb5ac142800 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3319407655 0 0) 0x7fb5ac13b060 con 0x7fb5ac13f970 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 >> v1:172.21.15.45:6790/0 conn(0x7fb5ac142800 legacy=0x7fb5ac1dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 >> v1:172.21.15.190:6789/0 conn(0x7fb5ac0ae250 legacy=0x7fb5ac144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 monclient: found mon.a 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac13c1d0 con 0x7fb5ac13f970 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:15.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 10 monclient: _renew_subs 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac13c4b0 con 0x7fb5ac13f970 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb59c001cf0 con 0x7fb5ac13f970 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fb59c003790 con 0x7fb5ac13f970 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5a27fc640 10 monclient: dump: 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:15.672 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.668+0000 7fb5b3e55640 10 monclient: _renew_subs 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.669+0000 7fb5b3e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.669+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac144e30 con 0x7fb5ac13f970 2024-03-29T11:37:15.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.669+0000 7fb5b3e55640 1 librados: init done 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.669+0000 7fb5a37fe640 4 set_mon_vals no callback set 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.670+0000 7fb5b3e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.670+0000 7fb5b3e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.671+0000 7fb5b3e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.671+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594005080 con 0x7fb5ac13f970 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.671+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fb59c0038f0 con 0x7fb5ac13f970 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.672+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fb59c0837f0 con 0x7fb5ac13f970 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.672+0000 7fb5a27fc640 10 client.4410.objecter ms_dispatch 0x7fb5ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.672+0000 7fb5a27fc640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:15.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.672+0000 7fb5a27fc640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.672+0000 7fb5a27fc640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:37:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.674+0000 7fb5b1bf1640 10 monclient: get_auth_request con 0x7fb59006df10 auth_method 0 2024-03-29T11:37:15.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.674+0000 7fb5a27fc640 10 client.4410.objecter ms_handle_connect 0x7fb59006df10 2024-03-29T11:37:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.680+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb59c04ef00 con 0x7fb5ac13f970 2024-03-29T11:37:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.680+0000 7fb5a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:15.680+0000 7fb5a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:16.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.014+0000 7fb5b3e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.014+0000 7fb5b3e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.014+0000 7fb5b3e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:16.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.014+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb594005330 con 0x7fb5ac13f970 2024-03-29T11:37:16.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.020+0000 7fb5a27fc640 1 -- 172.21.15.45:0/950941317 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fb59c051000 con 0x7fb5ac13f970 2024-03-29T11:37:16.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.020+0000 7fb5a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:16.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.020+0000 7fb5a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:16.019 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:16.019 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.023+0000 7fb5b3e55640 10 librados: watch_flush enter 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.023+0000 7fb5b3e55640 10 librados: watch_flush exit 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.023+0000 7fb5b3e55640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.023+0000 7fb5b3e55640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 >> v1:172.21.15.190:6816/1411598462 conn(0x7fb59006df10 legacy=0x7fb5900703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 10 monclient: shutdown 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 >> v1:172.21.15.45:6789/0 conn(0x7fb5ac13f970 legacy=0x7fb5ac1db5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 shutdown_connections 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.026+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 >> 172.21.15.45:0/950941317 conn(0x7fb5ac0b35e0 msgr2=0x7fb5ac0b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.027+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 shutdown_connections 2024-03-29T11:37:16.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.027+0000 7fb5b3e55640 1 -- 172.21.15.45:0/950941317 wait complete. 2024-03-29T11:37:16.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:16.027+0000 7fb5b3e55640 1 librados: shutdown 2024-03-29T11:37:16.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:22.851 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:23.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.001+0000 7fc6e88a7640 10 monclient: get_monmap_and_config 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.001+0000 7fc6e88a7640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.001+0000 7fc6e88a7640 10 monclient: monmap: 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:23.002466+0000 2024-03-29T11:37:23.002 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:23.002466+0000 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:23.003 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.002+0000 7fc6e88a7640 1 Processor -- start 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.002+0000 7fc6e88a7640 1 -- start start 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.002+0000 7fc6e88a7640 10 monclient: init 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.003+0000 7fc6e88a7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.003+0000 7fc6e88a7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.003+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc6dc006890 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:23.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.003+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc6dc006c70 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc6dc02b4c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc02c830 con 0x7fc6dc006890 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc02afa0 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc02aba0 con 0x7fc6dc006c70 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.006+0000 7fc6e88a7640 10 monclient(hunting): authenticate will time out at 1099.246027s 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e359e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc6dc02b4c0 0x7fc6dc0cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:58969/0 (socket says 172.21.15.45:58969) 2024-03-29T11:37:23.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e359e640 1 -- 172.21.15.45:0/1115966331 learned_addr learned my addr 172.21.15.45:0/1115966331 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602794033 0 0) 0x7fc6dc02afa0 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:37:23.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.008+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d0005730 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 957439635 0 0) 0x7fc6d0005730 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 >> v1:172.21.15.45:6790/0 conn(0x7fc6dc006c70 legacy=0x7fc6dc0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:23.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 >> v1:172.21.15.45:6789/0 conn(0x7fc6dc006890 legacy=0x7fc6dc0c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:23.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc02ad30 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc6cc001690 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.009+0000 7fc6e1d9b640 10 monclient: dump: 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.011 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.010+0000 7fc6e88a7640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:37:23.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.010+0000 7fc6e88a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:23.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6cc0017f0 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 1 -- 172.21.15.45:0/1115966331 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc6cc001c40 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e1d9b640 10 monclient: dump: 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:23.014 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e88a7640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.012+0000 7fc6e88a7640 4 set_mon_vals no callback set 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.014+0000 7fc6e88a7640 10 monclient: shutdown 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.014+0000 7fc6e88a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.014+0000 7fc6e88a7640 1 -- 172.21.15.45:0/1115966331 >> v1:172.21.15.190:6789/0 conn(0x7fc6dc02b4c0 legacy=0x7fc6dc0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.014+0000 7fc6e88a7640 1 -- 172.21.15.45:0/1115966331 shutdown_connections 2024-03-29T11:37:23.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.014+0000 7fc6e88a7640 1 -- 172.21.15.45:0/1115966331 >> 172.21.15.45:0/1115966331 conn(0x7fc6dc0058f0 msgr2=0x7fc6dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:23.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.015+0000 7fc6e88a7640 1 -- 172.21.15.45:0/1115966331 shutdown_connections 2024-03-29T11:37:23.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.015+0000 7fc6e88a7640 1 -- 172.21.15.45:0/1115966331 wait complete. 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.016+0000 7fc6e88a7640 10 monclient: build_initial_monmap 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.016+0000 7fc6e88a7640 10 monclient: monmap: 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:23.017 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:23.018 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 librados: starting msgr at 2024-03-29T11:37:23.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 librados: starting objecter 2024-03-29T11:37:23.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 Processor -- start 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 -- start start 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 librados: setting wanted keys 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 1 librados: calling monclient init 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.017+0000 7fc6e88a7640 10 monclient: init 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.018+0000 7fc6e88a7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:23.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.018+0000 7fc6e88a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:23.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc6dc006890 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:23.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc6dc02b4c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:23.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc6dc166ee0 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0c21d0 con 0x7fc6dc006890 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0bc410 con 0x7fc6dc02b4c0 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0baa30 con 0x7fc6dc166ee0 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e88a7640 10 monclient(hunting): authenticate will time out at 1099.260557s 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e2d9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc6dc006890 0x7fc6dc0ba320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56527/0 (socket says 172.21.15.45:56527) 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.021+0000 7fc6e2d9d640 1 -- 172.21.15.45:0/3923985490 learned_addr learned my addr 172.21.15.45:0/3923985490 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.024+0000 7fc6db7fe640 10 client.?.objecter ms_handle_connect 0x7fc6dc006890 2024-03-29T11:37:23.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.024+0000 7fc6db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909173280 0 0) 0x7fc6dc0c21d0 con 0x7fc6dc006890 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c4004eb0 con 0x7fc6dc006890 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 client.?.objecter ms_handle_connect 0x7fc6dc166ee0 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560419864 0 0) 0x7fc6dc0baa30 con 0x7fc6dc166ee0 2024-03-29T11:37:23.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c40051d0 con 0x7fc6dc166ee0 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.026+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878206429 0 0) 0x7fc6c40051d0 con 0x7fc6dc166ee0 2024-03-29T11:37:23.027 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6dc0baa30 con 0x7fc6dc166ee0 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc6cc001b00 con 0x7fc6dc166ee0 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:23.028 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158995189 0 0) 0x7fc6c4004eb0 con 0x7fc6dc006890 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c40051d0 con 0x7fc6dc006890 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc6d4003fa0 con 0x7fc6dc006890 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.027+0000 7fc6db7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:23.029 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666711705 0 0) 0x7fc6c40051d0 con 0x7fc6dc006890 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 >> v1:172.21.15.45:6790/0 conn(0x7fc6dc166ee0 legacy=0x7fc6dc0bd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 >> v1:172.21.15.190:6789/0 conn(0x7fc6dc02b4c0 legacy=0x7fc6dc1667d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 monclient: found mon.a 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0bced0 con 0x7fc6dc006890 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:23.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6d40044b0 con 0x7fc6dc006890 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.028+0000 7fc6e0d99640 4 set_mon_vals no callback set 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc6d40010b0 con 0x7fc6dc006890 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:23.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6db7fe640 10 monclient: dump: 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6e88a7640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6e88a7640 10 monclient: _renew_subs 2024-03-29T11:37:23.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.029+0000 7fc6e88a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc0bcb60 con 0x7fc6dc006890 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 10 monclient: _renew_subs 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc0bb320 con 0x7fc6dc006890 2024-03-29T11:37:23.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.030+0000 7fc6e88a7640 1 librados: init done 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.034+0000 7fc6e88a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.034+0000 7fc6e88a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.034+0000 7fc6e88a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.034+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc0bd0c0 con 0x7fc6dc006890 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.035+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fc6d4001290 con 0x7fc6dc006890 2024-03-29T11:37:23.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6e259c640 10 monclient: get_auth_request con 0x7fc6c406df90 auth_method 0 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fc6d4054770 con 0x7fc6dc006890 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 10 client.4413.objecter ms_dispatch 0x7fc6e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:37:23.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.036+0000 7fc6db7fe640 10 client.4413.objecter ms_handle_connect 0x7fc6c406df90 2024-03-29T11:37:23.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.048+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6d4050a10 con 0x7fc6dc006890 2024-03-29T11:37:23.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.048+0000 7fc6db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:23.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.048+0000 7fc6db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.394+0000 7fc6e88a7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.394+0000 7fc6e88a7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.394+0000 7fc6e88a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.394+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6dc02a520 con 0x7fc6dc006890 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.395+0000 7fc6db7fe640 1 -- 172.21.15.45:0/3923985490 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fc6d4050a10 con 0x7fc6dc006890 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.395+0000 7fc6db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.395+0000 7fc6db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:23.398 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:23.398 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.398+0000 7fc6e88a7640 10 librados: watch_flush enter 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 10 librados: watch_flush exit 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 >> v1:172.21.15.190:6816/1411598462 conn(0x7fc6c406df90 legacy=0x7fc6c4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 10 monclient: shutdown 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 >> v1:172.21.15.45:6789/0 conn(0x7fc6dc006890 legacy=0x7fc6dc0ba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 shutdown_connections 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 >> 172.21.15.45:0/3923985490 conn(0x7fc6dc0058f0 msgr2=0x7fc6dc038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 shutdown_connections 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.399+0000 7fc6e88a7640 1 -- 172.21.15.45:0/3923985490 wait complete. 2024-03-29T11:37:23.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:23.400+0000 7fc6e88a7640 1 librados: shutdown 2024-03-29T11:37:23.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:30.225 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:30.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.374+0000 7fe3324c3640 10 monclient: get_monmap_and_config 2024-03-29T11:37:30.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.374+0000 7fe3324c3640 10 monclient: build_initial_monmap 2024-03-29T11:37:30.375 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.374+0000 7fe3324c3640 10 monclient: monmap: 2024-03-29T11:37:30.375 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:30.375 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:30.375661+0000 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:30.375661+0000 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:30.376 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.375+0000 7fe3324c3640 1 Processor -- start 2024-03-29T11:37:30.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.375+0000 7fe3324c3640 1 -- start start 2024-03-29T11:37:30.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.375+0000 7fe3324c3640 10 monclient: init 2024-03-29T11:37:30.379 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-b con 0x7fe32c0a9930 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-c con 0x7fe32c149dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-a con 0x7fe32c14ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0aae60 con 0x7fe32c14ff40 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0b4310 con 0x7fe32c149dc0 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0afdd0 con 0x7fe32c0a9930 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.376+0000 7fe3324c3640 10 monclient(hunting): authenticate will time out at 1106.615893s 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.379+0000 7fe330a60640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe32c14ff40 0x7fe32c152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41616/0 (socket says 172.21.15.45:41616) 2024-03-29T11:37:30.380 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.379+0000 7fe330a60640 1 -- 172.21.15.45:0/2903037490 learned_addr learned my addr 172.21.15.45:0/2903037490 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838572604 0 0) 0x7fe32c0aae60 con 0x7fe32c14ff40 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.380+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314003520 con 0x7fe32c14ff40 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899046523 0 0) 0x7fe32c0b4310 con 0x7fe32c149dc0 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.385 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32c0aae60 con 0x7fe32c149dc0 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816357127 0 0) 0x7fe314003520 con 0x7fe32c14ff40 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 >> v1:172.21.15.45:6790/0 conn(0x7fe32c0a9930 legacy=0x7fe32c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 >> v1:172.21.15.190:6789/0 conn(0x7fe32c149dc0 legacy=0x7fe32c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 monclient: found mon.noname-a 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c14a950 con 0x7fe32c14ff40 2024-03-29T11:37:30.386 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient: _finish_auth 0 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.381+0000 7fe32affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe3324c3640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe3324c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe31c001610 con 0x7fe32c14ff40 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe32affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe32affd640 10 monclient: dump: 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.387 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.382+0000 7fe3324c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.383+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe31c001830 con 0x7fe32c14ff40 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.383+0000 7fe32affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.383+0000 7fe3324c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.383+0000 7fe3324c3640 4 set_mon_vals no callback set 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe32affd640 1 -- 172.21.15.45:0/2903037490 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe31c001bc0 con 0x7fe32c14ff40 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.388 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe32affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe32affd640 10 monclient: dump: 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.389 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe3324c3640 10 monclient: shutdown 2024-03-29T11:37:30.390 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe3324c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:30.390 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.384+0000 7fe3324c3640 1 -- 172.21.15.45:0/2903037490 >> v1:172.21.15.45:6789/0 conn(0x7fe32c14ff40 legacy=0x7fe32c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.385+0000 7fe3324c3640 1 -- 172.21.15.45:0/2903037490 shutdown_connections 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.386+0000 7fe3324c3640 1 -- 172.21.15.45:0/2903037490 >> 172.21.15.45:0/2903037490 conn(0x7fe32c0b35e0 msgr2=0x7fe32c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.388+0000 7fe3324c3640 1 -- 172.21.15.45:0/2903037490 shutdown_connections 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.388+0000 7fe3324c3640 1 -- 172.21.15.45:0/2903037490 wait complete. 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.389+0000 7fe3324c3640 10 monclient: build_initial_monmap 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.389+0000 7fe3324c3640 10 monclient: monmap: 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:30.391 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 librados: starting msgr at 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 librados: starting objecter 2024-03-29T11:37:30.392 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 Processor -- start 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 -- start start 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 librados: setting wanted keys 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 1 librados: calling monclient init 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.390+0000 7fe3324c3640 10 monclient: init 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.391+0000 7fe3324c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.391+0000 7fe3324c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-c con 0x7fe32c0a9930 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-b con 0x7fe32c149dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 10 monclient(hunting): picked mon.noname-a con 0x7fe32c14ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c14aae0 con 0x7fe32c14ff40 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c1313a0 con 0x7fe32c149dc0 2024-03-29T11:37:30.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c13f6c0 con 0x7fe32c0a9930 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe3324c3640 10 monclient(hunting): authenticate will time out at 1106.632471s 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe330a60640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe32c14ff40 0x7fe32c145450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43610/0 (socket says 172.21.15.45:43610) 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe330a60640 1 -- 172.21.15.45:0/2047173106 learned_addr learned my addr 172.21.15.45:0/2047173106 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe328ff9640 10 client.?.objecter ms_handle_connect 0x7fe32c14ff40 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.393+0000 7fe328ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503952683 0 0) 0x7fe32c14aae0 con 0x7fe32c14ff40 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.394+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe318004eb0 con 0x7fe32c14ff40 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2938955577 0 0) 0x7fe318004eb0 con 0x7fe32c14ff40 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe32c14aae0 con 0x7fe32c14ff40 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe31c001b40 con 0x7fe32c14ff40 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.397 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.396+0000 7fe328ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 client.?.objecter ms_handle_connect 0x7fe32c149dc0 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2331875106 0 0) 0x7fe32c14aae0 con 0x7fe32c14ff40 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 >> v1:172.21.15.45:6790/0 conn(0x7fe32c0a9930 legacy=0x7fe32c131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 >> v1:172.21.15.190:6789/0 conn(0x7fe32c149dc0 legacy=0x7fe32c1d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 monclient: found mon.a 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c146ba0 con 0x7fe32c14ff40 2024-03-29T11:37:30.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 10 monclient: _renew_subs 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c146e10 con 0x7fe32c14ff40 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe31c015b30 con 0x7fe32c14ff40 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 10 monclient: _renew_subs 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c131f40 con 0x7fe32c14ff40 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe3324c3640 1 librados: init done 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe31c015e40 con 0x7fe32c14ff40 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe328ff9640 10 monclient: dump: 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:30.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.398+0000 7fe32a7fc640 4 set_mon_vals no callback set 2024-03-29T11:37:30.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.401+0000 7fe3324c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.401+0000 7fe3324c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.401+0000 7fe3324c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.401+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c005080 con 0x7fe32c14ff40 2024-03-29T11:37:30.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.404+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe31c02be10 con 0x7fe32c14ff40 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.404+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe31c095b10 con 0x7fe32c14ff40 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.404+0000 7fe328ff9640 10 client.4419.objecter ms_dispatch 0x7fe32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.404+0000 7fe328ff9640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.404+0000 7fe328ff9640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.405+0000 7fe328ff9640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.405+0000 7fe32bfff640 10 monclient: get_auth_request con 0x7fe31806dbd0 auth_method 0 2024-03-29T11:37:30.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.405+0000 7fe328ff9640 10 client.4419.objecter ms_handle_connect 0x7fe31806dbd0 2024-03-29T11:37:30.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.416+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe31c0613f0 con 0x7fe32c14ff40 2024-03-29T11:37:30.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.416+0000 7fe328ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.417 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.416+0000 7fe328ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:30.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.781+0000 7fe3324c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.781+0000 7fe3324c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.781+0000 7fe3324c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:37:30.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.781+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe30c005330 con 0x7fe32c14ff40 2024-03-29T11:37:30.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.783+0000 7fe328ff9640 1 -- 172.21.15.45:0/2047173106 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe31c063320 con 0x7fe32c14ff40 2024-03-29T11:37:30.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.783+0000 7fe328ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.783+0000 7fe328ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:30.786 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:30.786 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:30.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.790+0000 7fe3324c3640 10 librados: watch_flush enter 2024-03-29T11:37:30.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.790+0000 7fe3324c3640 10 librados: watch_flush exit 2024-03-29T11:37:30.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.790+0000 7fe3324c3640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:37:30.790 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.790+0000 7fe3324c3640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:37:30.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.792+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe31806dbd0 legacy=0x7fe318070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.792+0000 7fe3324c3640 10 monclient: shutdown 2024-03-29T11:37:30.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.792+0000 7fe3324c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:30.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.792+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 >> v1:172.21.15.45:6789/0 conn(0x7fe32c14ff40 legacy=0x7fe32c145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.793+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 shutdown_connections 2024-03-29T11:37:30.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.793+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 >> 172.21.15.45:0/2047173106 conn(0x7fe32c0b35e0 msgr2=0x7fe32c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:30.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.794+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 shutdown_connections 2024-03-29T11:37:30.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.794+0000 7fe3324c3640 1 -- 172.21.15.45:0/2047173106 wait complete. 2024-03-29T11:37:30.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:30.794+0000 7fe3324c3640 1 librados: shutdown 2024-03-29T11:37:30.815 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:37.623 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.759+0000 7fe54424a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.759+0000 7fe54424a640 10 monclient: build_initial_monmap 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.759+0000 7fe54424a640 10 monclient: monmap: 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:37.760270+0000 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:37.760270+0000 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:37.759 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.760 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:37.760 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:37.760 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:37.760 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.759+0000 7fe54424a640 1 Processor -- start 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.760+0000 7fe54424a640 1 -- start start 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.760+0000 7fe54424a640 10 monclient: init 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.760+0000 7fe54424a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c13f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:37.761 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:37.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0aaef0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c145bd0 con 0x7fe53c13f650 2024-03-29T11:37:37.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c145eb0 con 0x7fe53c1457f0 2024-03-29T11:37:37.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.762 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.761+0000 7fe54424a640 10 monclient(hunting): authenticate will time out at 1114.000482s 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe541fe6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe53c1457f0 0x7fe53c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46410/0 (socket says 172.21.15.45:46410) 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe541fe6640 1 -- 172.21.15.45:0/193262816 learned_addr learned my addr 172.21.15.45:0/193262816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951719412 0 0) 0x7fe53c145eb0 con 0x7fe53c1457f0 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.762+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528003520 con 0x7fe53c1457f0 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.763+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978956106 0 0) 0x7fe528003520 con 0x7fe53c1457f0 2024-03-29T11:37:37.763 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.763+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 >> v1:172.21.15.190:6789/0 conn(0x7fe53c13f650 legacy=0x7fe53c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 >> v1:172.21.15.45:6789/0 conn(0x7fe53c0a9d70 legacy=0x7fe53c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 monclient: found mon.noname-b 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c146340 con 0x7fe53c1457f0 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.765 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe538001690 con 0x7fe53c1457f0 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: dump: 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe54424a640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:37:37.766 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe54424a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe538001860 con 0x7fe53c1457f0 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe540fe4640 1 -- 172.21.15.45:0/193262816 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe538001cb0 con 0x7fe53c1457f0 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe54424a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.765+0000 7fe54424a640 4 set_mon_vals no callback set 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.766+0000 7fe540fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.766+0000 7fe540fe4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.766+0000 7fe540fe4640 10 monclient: dump: 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:37.768 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.766+0000 7fe54424a640 10 monclient: shutdown 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.767+0000 7fe54424a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.767+0000 7fe54424a640 1 -- 172.21.15.45:0/193262816 >> v1:172.21.15.45:6790/0 conn(0x7fe53c1457f0 legacy=0x7fe53c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.767+0000 7fe54424a640 1 -- 172.21.15.45:0/193262816 shutdown_connections 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.767+0000 7fe54424a640 1 -- 172.21.15.45:0/193262816 >> 172.21.15.45:0/193262816 conn(0x7fe53c066a00 msgr2=0x7fe53c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.768+0000 7fe54424a640 1 -- 172.21.15.45:0/193262816 shutdown_connections 2024-03-29T11:37:37.769 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.768+0000 7fe54424a640 1 -- 172.21.15.45:0/193262816 wait complete. 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.769+0000 7fe54424a640 10 monclient: build_initial_monmap 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.769+0000 7fe54424a640 10 monclient: monmap: 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:37.771 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 librados: starting msgr at 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 librados: starting objecter 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 Processor -- start 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 -- start start 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 librados: setting wanted keys 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 1 librados: calling monclient init 2024-03-29T11:37:37.772 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.770+0000 7fe54424a640 10 monclient: init 2024-03-29T11:37:37.778 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.771+0000 7fe54424a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.771+0000 7fe54424a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.771+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c0a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe541fe6640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe53c0a9d70 0x7fe53c1d7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:50966/0 (socket says 172.21.15.45:50966) 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe541fe6640 1 -- 172.21.15.45:0/1757430816 learned_addr learned my addr 172.21.15.45:0/1757430816 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe5327fc640 10 client.?.objecter ms_handle_connect 0x7fe53c0a9d70 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe5327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c13f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe54424a640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.772+0000 7fe5327fc640 10 client.?.objecter ms_handle_connect 0x7fe53c13f650 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe54424a640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0acaf0 con 0x7fe53c13f650 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe54424a640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c142580 con 0x7fe53c1457f0 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe54424a640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c142860 con 0x7fe53c0a9d70 2024-03-29T11:37:37.779 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe54424a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe54424a640 10 monclient(hunting): authenticate will time out at 1114.012176s 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008746129 0 0) 0x7fe53c142860 con 0x7fe53c0a9d70 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe520004eb0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921320184 0 0) 0x7fe53c0acaf0 con 0x7fe53c13f650 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.773+0000 7fe5327fc640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c142860 con 0x7fe53c13f650 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991540245 0 0) 0x7fe520004eb0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.780 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe53c0acaf0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe538001bb0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 10 monclient(hunting): dump: 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:37.781 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3627852376 0 0) 0x7fe53c0acaf0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.774+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 >> v1:172.21.15.190:6789/0 conn(0x7fe53c1457f0 legacy=0x7fe53c1e2cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 >> v1:172.21.15.45:6789/0 conn(0x7fe53c13f650 legacy=0x7fe53c136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 1 monclient: found mon.c 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c142d80 con 0x7fe53c0a9d70 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.776+0000 7fe5327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe54424a640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:37:37.782 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe538004a90 con 0x7fe53c0a9d70 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe538004da0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe5327fc640 10 monclient: dump: 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:37.783 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe54424a640 10 monclient: _renew_subs 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe54424a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe54424a640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c141d10 con 0x7fe53c0a9d70 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.777+0000 7fe533fff640 4 set_mon_vals no callback set 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.778+0000 7fe54424a640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.778+0000 7fe54424a640 10 monclient: _renew_subs 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.778+0000 7fe54424a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.778+0000 7fe54424a640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c0ab800 con 0x7fe53c0a9d70 2024-03-29T11:37:37.784 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.778+0000 7fe54424a640 1 librados: init done 2024-03-29T11:37:37.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.780+0000 7fe51ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.780+0000 7fe51ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.780+0000 7fe51ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.780+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c1429f0 con 0x7fe53c0a9d70 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.780+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe53801ad70 con 0x7fe53c0a9d70 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.781+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe538084b20 con 0x7fe53c0a9d70 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.781+0000 7fe5327fc640 10 client.4445.objecter ms_dispatch 0x7fe53c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.781+0000 7fe5327fc640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.781+0000 7fe5327fc640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.781+0000 7fe5327fc640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.783+0000 7fe5417e5640 10 monclient: get_auth_request con 0x7fe52006dad0 auth_method 0 2024-03-29T11:37:37.786 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.784+0000 7fe5327fc640 10 client.4445.objecter ms_handle_connect 0x7fe52006dad0 2024-03-29T11:37:37.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.788+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe538050400 con 0x7fe53c0a9d70 2024-03-29T11:37:37.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.788+0000 7fe5327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:37.788+0000 7fe5327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:38.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.120+0000 7fe51ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.120+0000 7fe51ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.120+0000 7fe51ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:38.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.120+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe53c0aada0 con 0x7fe53c0a9d70 2024-03-29T11:37:38.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.123+0000 7fe5327fc640 1 -- 172.21.15.45:0/1757430816 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe538052330 con 0x7fe53c0a9d70 2024-03-29T11:37:38.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.123+0000 7fe5327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:38.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.123+0000 7fe5327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:38.124 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:38.124 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.126+0000 7fe51ffff640 10 librados: watch_flush enter 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.126+0000 7fe51ffff640 10 librados: watch_flush exit 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.126+0000 7fe51ffff640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.126+0000 7fe51ffff640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.127+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe52006dad0 legacy=0x7fe52006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.127+0000 7fe51ffff640 10 monclient: shutdown 2024-03-29T11:37:38.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.127+0000 7fe51ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:38.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.127+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 >> v1:172.21.15.45:6790/0 conn(0x7fe53c0a9d70 legacy=0x7fe53c1d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:38.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.128+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 shutdown_connections 2024-03-29T11:37:38.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.128+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 >> 172.21.15.45:0/1757430816 conn(0x7fe53c066a00 msgr2=0x7fe53c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:38.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.128+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 shutdown_connections 2024-03-29T11:37:38.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.128+0000 7fe51ffff640 1 -- 172.21.15.45:0/1757430816 wait complete. 2024-03-29T11:37:38.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:38.128+0000 7fe51ffff640 1 librados: shutdown 2024-03-29T11:37:38.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:44.956 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:45.102 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.102+0000 7f1d85b6e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.102+0000 7f1d85b6e640 10 monclient: build_initial_monmap 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.102+0000 7f1d85b6e640 10 monclient: monmap: 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:45.103513+0000 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:45.103513+0000 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:45.103 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:45.104 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.103+0000 7f1d85b6e640 1 Processor -- start 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.103+0000 7f1d85b6e640 1 -- start start 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.103+0000 7f1d85b6e640 10 monclient: init 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:45.105 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-c con 0x7f1d80143a90 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-a con 0x7f1d80146fd0 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-b con 0x7f1d800b2e50 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8006ef90 con 0x7f1d80146fd0 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8006f1c0 con 0x7f1d80143a90 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d80071eb0 con 0x7f1d800b2e50 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d85b6e640 10 monclient(hunting): authenticate will time out at 1121.343487s 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d84b6c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f1d80143a90 0x7f1d80146320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:59616/0 (socket says 172.21.15.45:59616) 2024-03-29T11:37:45.106 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.104+0000 7f1d84b6c640 1 -- 172.21.15.45:0/1647312969 learned_addr learned my addr 172.21.15.45:0/1647312969 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541092841 0 0) 0x7f1d8006f1c0 con 0x7f1d80143a90 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d70003520 con 0x7f1d80143a90 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260181739 0 0) 0x7f1d80071eb0 con 0x7f1d800b2e50 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:37:45.108 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8006f1c0 con 0x7f1d800b2e50 2024-03-29T11:37:45.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579753936 0 0) 0x7f1d70003520 con 0x7f1d80143a90 2024-03-29T11:37:45.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 >> v1:172.21.15.45:6790/0 conn(0x7f1d800b2e50 legacy=0x7f1d80151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 >> v1:172.21.15.45:6789/0 conn(0x7f1d80146fd0 legacy=0x7f1d800b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800b3300 con 0x7f1d80143a90 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1d74001690 con 0x7f1d80143a90 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.105+0000 7f1d7f7fe640 10 monclient: dump: 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.112 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d85b6e640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d85b6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d740017f0 con 0x7f1d80143a90 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 1 -- 172.21.15.45:0/1647312969 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1d74001c00 con 0x7f1d80143a90 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d7f7fe640 10 monclient: dump: 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:45.113 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d85b6e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.106+0000 7f1d85b6e640 4 set_mon_vals no callback set 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.107+0000 7f1d85b6e640 10 monclient: shutdown 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.107+0000 7f1d85b6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.107+0000 7f1d85b6e640 1 -- 172.21.15.45:0/1647312969 >> v1:172.21.15.190:6789/0 conn(0x7f1d80143a90 legacy=0x7f1d80146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.107+0000 7f1d85b6e640 1 -- 172.21.15.45:0/1647312969 shutdown_connections 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.107+0000 7f1d85b6e640 1 -- 172.21.15.45:0/1647312969 >> 172.21.15.45:0/1647312969 conn(0x7f1d8006d0e0 msgr2=0x7f1d80066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.108+0000 7f1d85b6e640 1 -- 172.21.15.45:0/1647312969 shutdown_connections 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.108+0000 7f1d85b6e640 1 -- 172.21.15.45:0/1647312969 wait complete. 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.109+0000 7f1d85b6e640 10 monclient: build_initial_monmap 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.109+0000 7f1d85b6e640 10 monclient: monmap: 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:45.115 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:45.116 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.109+0000 7f1d85b6e640 1 librados: starting msgr at 2024-03-29T11:37:45.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.109+0000 7f1d85b6e640 1 librados: starting objecter 2024-03-29T11:37:45.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.110+0000 7f1d85b6e640 1 Processor -- start 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.110+0000 7f1d85b6e640 1 -- start start 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.110+0000 7f1d85b6e640 1 librados: setting wanted keys 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.110+0000 7f1d85b6e640 1 librados: calling monclient init 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.110+0000 7f1d85b6e640 10 monclient: init 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.111+0000 7f1d85b6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.111+0000 7f1d85b6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.111+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-a con 0x7f1d800b2e50 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.111+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-c con 0x7f1d80143a90 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d84b6c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f1d800b2e50 0x7f1d801d85e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:59960/0 (socket says 172.21.15.45:59960) 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d84b6c640 1 -- 172.21.15.45:0/4025313350 learned_addr learned my addr 172.21.15.45:0/4025313350 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 10 monclient(hunting): picked mon.noname-b con 0x7f1d80146fd0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:45.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d801d8cf0 con 0x7f1d800b2e50 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8013d9e0 con 0x7f1d80146fd0 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d801523f0 con 0x7f1d80143a90 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d85b6e640 10 monclient(hunting): authenticate will time out at 1121.351702s 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f1d80146fd0 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f1d800b2e50 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter ms_handle_connect 0x7f1d80143a90 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427833806 0 0) 0x7f1d801523f0 con 0x7f1d80143a90 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.112+0000 7f1d7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d64004eb0 con 0x7f1d80143a90 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939671662 0 0) 0x7f1d8013d9e0 con 0x7f1d80146fd0 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d801523f0 con 0x7f1d80146fd0 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3090808979 0 0) 0x7f1d801523f0 con 0x7f1d80146fd0 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d8013d9e0 con 0x7f1d80146fd0 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1d78001c70 con 0x7f1d80146fd0 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:45.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034189120 0 0) 0x7f1d801d8cf0 con 0x7f1d800b2e50 2024-03-29T11:37:45.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d801523f0 con 0x7f1d800b2e50 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090979487 0 0) 0x7f1d64004eb0 con 0x7f1d80143a90 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d801d8cf0 con 0x7f1d80143a90 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1d6c001890 con 0x7f1d80143a90 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.113+0000 7f1d7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.126 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607800017 0 0) 0x7f1d8013d9e0 con 0x7f1d80146fd0 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 >> v1:172.21.15.45:6790/0 conn(0x7f1d80143a90 legacy=0x7f1d8013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 >> v1:172.21.15.45:6789/0 conn(0x7f1d800b2e50 legacy=0x7f1d801d85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 monclient: found mon.b 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:45.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d80152580 con 0x7f1d80146fd0 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 10 monclient: _renew_subs 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8013d410 con 0x7f1d80146fd0 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 10 monclient: _renew_subs 2024-03-29T11:37:45.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d801dbd60 con 0x7f1d80146fd0 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d85b6e640 1 librados: init done 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d780020e0 con 0x7f1d80146fd0 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7effd640 4 set_mon_vals no callback set 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1d78003b30 con 0x7f1d80146fd0 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.114+0000 7f1d7d7fa640 10 monclient: dump: 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.129 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.116+0000 7f1d85b6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.116+0000 7f1d85b6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.116+0000 7f1d85b6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.116+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d60005080 con 0x7f1d80146fd0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.116+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f1d78019b80 con 0x7f1d80146fd0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f1d78083b90 con 0x7f1d80146fd0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d7d7fa640 10 client.4399.objecter ms_dispatch 0x7f1d80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d7d7fa640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d7d7fa640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d84b6c640 10 monclient: get_auth_request con 0x7f1d6406dfb0 auth_method 0 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.117+0000 7f1d7d7fa640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:37:45.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.118+0000 7f1d7d7fa640 10 client.4399.objecter ms_handle_connect 0x7f1d6406dfb0 2024-03-29T11:37:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.124+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d7804f470 con 0x7f1d80146fd0 2024-03-29T11:37:45.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.124+0000 7f1d7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:45.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.124+0000 7f1d7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.502+0000 7f1d85b6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.502+0000 7f1d85b6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.502+0000 7f1d85b6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.502+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d60005330 con 0x7f1d80146fd0 2024-03-29T11:37:45.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.503+0000 7f1d7d7fa640 1 -- 172.21.15.45:0/4025313350 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f1d780513a0 con 0x7f1d80146fd0 2024-03-29T11:37:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.503+0000 7f1d7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.504 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.503+0000 7f1d7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:45.504 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:45.504 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:45.506 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.506+0000 7f1d85b6e640 10 librados: watch_flush enter 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.506+0000 7f1d85b6e640 10 librados: watch_flush exit 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.506+0000 7f1d85b6e640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.506+0000 7f1d85b6e640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.506+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 >> v1:172.21.15.190:6816/1411598462 conn(0x7f1d6406dfb0 legacy=0x7f1d64070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 10 monclient: shutdown 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:45.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 >> v1:172.21.15.190:6789/0 conn(0x7f1d80146fd0 legacy=0x7f1d801db650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 shutdown_connections 2024-03-29T11:37:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 >> 172.21.15.45:0/4025313350 conn(0x7f1d8006d0e0 msgr2=0x7f1d8013f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 shutdown_connections 2024-03-29T11:37:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 -- 172.21.15.45:0/4025313350 wait complete. 2024-03-29T11:37:45.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:45.507+0000 7f1d85b6e640 1 librados: shutdown 2024-03-29T11:37:45.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:52.329 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.464+0000 7fdb71840640 10 monclient: get_monmap_and_config 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.464+0000 7fdb71840640 10 monclient: build_initial_monmap 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.464+0000 7fdb71840640 10 monclient: monmap: 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:52.466057+0000 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:52.466057+0000 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:52.465 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.466 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:52.466 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:52.466 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:52.466 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.465+0000 7fdb71840640 1 Processor -- start 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 1 -- start start 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient: init 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-c con 0x7fdb6c1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-b con 0x7fdb6c13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6c0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c0aaef0 con 0x7fdb6c0a9d70 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c145bd0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c145eb0 con 0x7fdb6c13f650 2024-03-29T11:37:52.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.466+0000 7fdb71840640 10 monclient(hunting): authenticate will time out at 1128.706079s 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdb6c1457f0 0x7fdb6c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:37852/0 (socket says 172.21.15.45:37852) 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6affd640 1 -- 172.21.15.45:0/1567229750 learned_addr learned my addr 172.21.15.45:0/1567229750 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb63fff640 1 --1- 172.21.15.45:0/1567229750 >> v1:172.21.15.45:6790/0 conn(0x7fdb6c13f650 0x7fdb6c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46876/0 (socket says 172.21.15.45:46876) 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207659070 0 0) 0x7fdb6c145eb0 con 0x7fdb6c13f650 2024-03-29T11:37:52.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb58003520 con 0x7fdb6c13f650 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4458014 0 0) 0x7fdb6c145bd0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb6c145eb0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343668591 0 0) 0x7fdb58003520 con 0x7fdb6c13f650 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 >> v1:172.21.15.190:6789/0 conn(0x7fdb6c1457f0 legacy=0x7fdb6c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.467+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 >> v1:172.21.15.45:6789/0 conn(0x7fdb6c0a9d70 legacy=0x7fdb6c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c146340 con 0x7fdb6c13f650 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb71840640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb71840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdb54001690 con 0x7fdb6c13f650 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:52.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: dump: 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb71840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb540017f0 con 0x7fdb6c13f650 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb71840640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:52.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.469+0000 7fdb71840640 4 set_mon_vals no callback set 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb6a7fc640 1 -- 172.21.15.45:0/1567229750 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdb54001c00 con 0x7fdb6c13f650 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb6a7fc640 10 monclient: dump: 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.473 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 10 monclient: shutdown 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 1 -- 172.21.15.45:0/1567229750 >> v1:172.21.15.45:6790/0 conn(0x7fdb6c13f650 legacy=0x7fdb6c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 1 -- 172.21.15.45:0/1567229750 shutdown_connections 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 1 -- 172.21.15.45:0/1567229750 >> 172.21.15.45:0/1567229750 conn(0x7fdb6c066a00 msgr2=0x7fdb6c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 1 -- 172.21.15.45:0/1567229750 shutdown_connections 2024-03-29T11:37:52.474 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.470+0000 7fdb71840640 1 -- 172.21.15.45:0/1567229750 wait complete. 2024-03-29T11:37:52.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.472+0000 7fdb71840640 10 monclient: build_initial_monmap 2024-03-29T11:37:52.476 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.472+0000 7fdb71840640 10 monclient: monmap: 2024-03-29T11:37:52.476 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:52.476 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:52.476 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 librados: starting msgr at 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 librados: starting objecter 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 Processor -- start 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 -- start start 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 librados: setting wanted keys 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 1 librados: calling monclient init 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.473+0000 7fdb71840640 10 monclient: init 2024-03-29T11:37:52.477 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6c0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-c con 0x7fdb6c13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient(hunting): picked mon.noname-b con 0x7fdb6c1457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c13bac0 con 0x7fdb6c0a9d70 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c06eba0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c138140 con 0x7fdb6c13f650 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb71840640 10 monclient(hunting): authenticate will time out at 1128.713339s 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb6b7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdb6c1457f0 0x7fdb6c137a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:38026/0 (socket says 172.21.15.45:38026) 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb6b7fe640 1 -- 172.21.15.45:0/3367885628 learned_addr learned my addr 172.21.15.45:0/3367885628 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 client.?.objecter ms_handle_connect 0x7fdb6c13f650 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:52.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 client.?.objecter ms_handle_connect 0x7fdb6c1457f0 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368653506 0 0) 0x7fdb6c138140 con 0x7fdb6c13f650 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c004eb0 con 0x7fdb6c13f650 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376316249 0 0) 0x7fdb6c06eba0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb6c138140 con 0x7fdb6c1457f0 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.474+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465881965 0 0) 0x7fdb4c004eb0 con 0x7fdb6c13f650 2024-03-29T11:37:52.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb6c06eba0 con 0x7fdb6c13f650 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdb54001bc0 con 0x7fdb6c13f650 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522254448 0 0) 0x7fdb6c138140 con 0x7fdb6c1457f0 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb4c004eb0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdb64001ce0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:52.481 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:52.482 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1447584977 0 0) 0x7fdb4c004eb0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 >> v1:172.21.15.45:6790/0 conn(0x7fdb6c13f650 legacy=0x7fdb6c137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.475+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 >> v1:172.21.15.45:6789/0 conn(0x7fdb6c0a9d70 legacy=0x7fdb6c1deb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.476+0000 7fdb68ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 1 monclient: found mon.b 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c13b5c0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:52.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 10 monclient: _renew_subs 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c06ee10 con 0x7fdb6c1457f0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 10 monclient: _renew_subs 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c142b10 con 0x7fdb6c1457f0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.477+0000 7fdb71840640 1 librados: init done 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb64003970 con 0x7fdb6c1457f0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdb64003dc0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb68ff9640 10 monclient: dump: 2024-03-29T11:37:52.485 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb697fa640 4 set_mon_vals no callback set 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.478+0000 7fdb71840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fdb64003f20 con 0x7fdb6c1457f0 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb71840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb71840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:52.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb50005080 con 0x7fdb6c1457f0 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fdb64084df0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.480+0000 7fdb68ff9640 10 client.4405.objecter ms_dispatch 0x7fdb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.481+0000 7fdb68ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.481+0000 7fdb68ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.481+0000 7fdb68ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.481+0000 7fdb6affd640 10 monclient: get_auth_request con 0x7fdb4c06dd40 auth_method 0 2024-03-29T11:37:52.487 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.481+0000 7fdb68ff9640 10 client.4405.objecter ms_handle_connect 0x7fdb4c06dd40 2024-03-29T11:37:52.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.489+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb640506d0 con 0x7fdb6c1457f0 2024-03-29T11:37:52.489 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.489+0000 7fdb68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.489+0000 7fdb68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.813+0000 7fdb71840640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.813+0000 7fdb71840640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.813 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.813+0000 7fdb71840640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:37:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.813+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb50005330 con 0x7fdb6c1457f0 2024-03-29T11:37:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.814+0000 7fdb68ff9640 1 -- 172.21.15.45:0/3367885628 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fdb64052600 con 0x7fdb6c1457f0 2024-03-29T11:37:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.814+0000 7fdb68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.814 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.814+0000 7fdb68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:52.815 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:37:52.815 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 10 librados: watch_flush enter 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 10 librados: watch_flush exit 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 >> v1:172.21.15.190:6816/1411598462 conn(0x7fdb4c06dd40 legacy=0x7fdb4c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 10 monclient: shutdown 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.817+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 >> v1:172.21.15.190:6789/0 conn(0x7fdb6c1457f0 legacy=0x7fdb6c137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.818+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 shutdown_connections 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.818+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 >> 172.21.15.45:0/3367885628 conn(0x7fdb6c066a00 msgr2=0x7fdb6c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.818+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 shutdown_connections 2024-03-29T11:37:52.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.818+0000 7fdb71840640 1 -- 172.21.15.45:0/3367885628 wait complete. 2024-03-29T11:37:52.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:52.818+0000 7fdb71840640 1 librados: shutdown 2024-03-29T11:37:52.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:59.639 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.782+0000 7f07b2443640 10 monclient: get_monmap_and_config 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.782+0000 7f07b2443640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.782+0000 7f07b2443640 10 monclient: monmap: 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:59.783 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:37:59.783613+0000 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:37:59.783613+0000 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:37:59.784 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.783+0000 7f07b2443640 1 Processor -- start 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.783+0000 7f07b2443640 1 -- start start 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.783+0000 7f07b2443640 10 monclient: init 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-b con 0x7f07ac0a9930 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-c con 0x7f07ac149dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-a con 0x7f07ac14ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0aae60 con 0x7f07ac14ff40 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0b4310 con 0x7f07ac149dc0 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0afdd0 con 0x7f07ac0a9930 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.785 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.784+0000 7f07b2443640 10 monclient(hunting): authenticate will time out at 1136.023662s 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.786+0000 7f07abfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07ac0a9930 0x7f07ac143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35810/0 (socket says 172.21.15.45:35810) 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.786+0000 7f07abfff640 1 -- 172.21.15.45:0/2459137519 learned_addr learned my addr 172.21.15.45:0/2459137519 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617664497 0 0) 0x7f07ac0afdd0 con 0x7f07ac0a9930 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:37:59.788 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.787+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0794003520 con 0x7f07ac0a9930 2024-03-29T11:37:59.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.788+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827911544 0 0) 0x7f0794003520 con 0x7f07ac0a9930 2024-03-29T11:37:59.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.788+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 >> v1:172.21.15.190:6789/0 conn(0x7f07ac149dc0 legacy=0x7f07ac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:59.789 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.788+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 >> v1:172.21.15.45:6789/0 conn(0x7f07ac14ff40 legacy=0x7f07ac152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 1 monclient: found mon.noname-b 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac14a950 con 0x7f07ac0a9930 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07b2443640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07b2443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f07a0001690 con 0x7f07ac0a9930 2024-03-29T11:37:59.791 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: dump: 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07b2443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07a0001920 con 0x7f07ac0a9930 2024-03-29T11:37:59.792 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07b2443640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07b2443640 4 set_mon_vals no callback set 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 1 -- 172.21.15.45:0/2459137519 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f07a0001cb0 con 0x7f07ac0a9930 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.789+0000 7f07aaffd640 10 monclient: dump: 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:59.793 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 10 monclient: shutdown 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 1 -- 172.21.15.45:0/2459137519 >> v1:172.21.15.45:6790/0 conn(0x7f07ac0a9930 legacy=0x7f07ac143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 1 -- 172.21.15.45:0/2459137519 shutdown_connections 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 1 -- 172.21.15.45:0/2459137519 >> 172.21.15.45:0/2459137519 conn(0x7f07ac0b35e0 msgr2=0x7f07ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 1 -- 172.21.15.45:0/2459137519 shutdown_connections 2024-03-29T11:37:59.794 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.790+0000 7f07b2443640 1 -- 172.21.15.45:0/2459137519 wait complete. 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.791+0000 7f07b2443640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.791+0000 7f07b2443640 10 monclient: monmap: 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.798 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.791+0000 7f07b2443640 1 librados: starting msgr at 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.791+0000 7f07b2443640 1 librados: starting objecter 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.791+0000 7f07b2443640 1 Processor -- start 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 1 -- start start 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 1 librados: setting wanted keys 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 1 librados: calling monclient init 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 10 monclient: init 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-a con 0x7f07ac149dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:37:59.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.792+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-c con 0x7f07ac14ff40 addr v1:172.21.15.45:6790/0 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 10 monclient(hunting): picked mon.noname-b con 0x7f07ac148cd0 addr v1:172.21.15.190:6789/0 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac140560 con 0x7f07ac149dc0 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac1407d0 con 0x7f07ac148cd0 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac13f5b0 con 0x7f07ac14ff40 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.793+0000 7f07b2443640 10 monclient(hunting): authenticate will time out at 1136.032147s 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.795+0000 7f07ab7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f07ac14ff40 0x7f07ac1db7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36260/0 (socket says 172.21.15.45:36260) 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.795+0000 7f07ab7fe640 1 -- 172.21.15.45:0/3055304960 learned_addr learned my addr 172.21.15.45:0/3055304960 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 client.?.objecter ms_handle_connect 0x7f07ac14ff40 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006507882 0 0) 0x7f07ac13f5b0 con 0x7f07ac14ff40 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.800 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0798004eb0 con 0x7f07ac14ff40 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 client.?.objecter ms_handle_connect 0x7f07ac148cd0 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227749261 0 0) 0x7f0798004eb0 con 0x7f07ac14ff40 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07ac13f5b0 con 0x7f07ac14ff40 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f07a4001ce0 con 0x7f07ac14ff40 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.801 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152652572 0 0) 0x7f07ac1407d0 con 0x7f07ac148cd0 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.796+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0798004eb0 con 0x7f07ac148cd0 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 client.?.objecter ms_handle_connect 0x7f07ac149dc0 2024-03-29T11:37:59.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013181823 0 0) 0x7f07ac140560 con 0x7f07ac149dc0 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac1407d0 con 0x7f07ac149dc0 2024-03-29T11:37:59.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158229056 0 0) 0x7f07ac13f5b0 con 0x7f07ac14ff40 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 >> v1:172.21.15.190:6789/0 conn(0x7f07ac148cd0 legacy=0x7f07ac1dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 >> v1:172.21.15.45:6789/0 conn(0x7f07ac149dc0 legacy=0x7f07ac13fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 monclient: found mon.c 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac13f740 con 0x7f07ac14ff40 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 10 monclient: _renew_subs 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac1e7200 con 0x7f07ac14ff40 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 10 monclient: _renew_subs 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac1e7630 con 0x7f07ac14ff40 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07b2443640 1 librados: init done 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07a4001fc0 con 0x7f07ac14ff40 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f07a4003dc0 con 0x7f07ac14ff40 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07a8ff9640 10 monclient: dump: 2024-03-29T11:37:59.808 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.797+0000 7f07aa7fc640 4 set_mon_vals no callback set 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.799+0000 7f07b2443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.799+0000 7f07b2443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.799+0000 7f07b2443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.799+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c005080 con 0x7f07ac14ff40 2024-03-29T11:37:59.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.799+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f07a4003f20 con 0x7f07ac14ff40 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.800+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f07a4084e80 con 0x7f07ac14ff40 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.800+0000 7f07a8ff9640 10 client.4463.objecter ms_dispatch 0x7f07ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.800+0000 7f07a8ff9640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.800+0000 7f07a8ff9640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.800+0000 7f07a8ff9640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.801+0000 7f07abfff640 10 monclient: get_auth_request con 0x7f079806dea0 auth_method 0 2024-03-29T11:37:59.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.803+0000 7f07a8ff9640 10 client.4463.objecter ms_handle_connect 0x7f079806dea0 2024-03-29T11:37:59.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.808+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07a4050760 con 0x7f07ac14ff40 2024-03-29T11:37:59.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.808+0000 7f07a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:37:59.808+0000 7f07a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:00.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.144+0000 7f07b2443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07b2443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07b2443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:00.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f078c005330 con 0x7f07ac14ff40 2024-03-29T11:38:00.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07a8ff9640 1 -- 172.21.15.45:0/3055304960 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f07a4052690 con 0x7f07ac14ff40 2024-03-29T11:38:00.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:00.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.145+0000 7f07a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:00.147 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:00.147 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:00.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.149+0000 7f07b2443640 10 librados: watch_flush enter 2024-03-29T11:38:00.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.149+0000 7f07b2443640 10 librados: watch_flush exit 2024-03-29T11:38:00.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.149+0000 7f07b2443640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:38:00.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.149+0000 7f07b2443640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:38:00.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 >> v1:172.21.15.190:6816/1411598462 conn(0x7f079806dea0 legacy=0x7f0798070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 10 monclient: shutdown 2024-03-29T11:38:00.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:00.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 >> v1:172.21.15.45:6790/0 conn(0x7f07ac14ff40 legacy=0x7f07ac1db7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:00.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 shutdown_connections 2024-03-29T11:38:00.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.151+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 >> 172.21.15.45:0/3055304960 conn(0x7f07ac0b35e0 msgr2=0x7f07ac0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:00.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.152+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 shutdown_connections 2024-03-29T11:38:00.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.152+0000 7f07b2443640 1 -- 172.21.15.45:0/3055304960 wait complete. 2024-03-29T11:38:00.152 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:00.152+0000 7f07b2443640 1 librados: shutdown 2024-03-29T11:38:00.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:06.976 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:07.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.121+0000 7fdbcd255640 10 monclient: get_monmap_and_config 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.121+0000 7fdbcd255640 10 monclient: build_initial_monmap 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.121+0000 7fdbcd255640 10 monclient: monmap: 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:07.122493+0000 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:07.122493+0000 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:07.122 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.122+0000 7fdbcd255640 1 Processor -- start 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.122+0000 7fdbcd255640 1 -- start start 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.122+0000 7fdbcd255640 10 monclient: init 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.122+0000 7fdbcd255640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:07.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc81457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc80a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc80aaef0 con 0x7fdbc813f650 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc8145bd0 con 0x7fdbc81457f0 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc8145eb0 con 0x7fdbc80a9d70 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:07.124 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.123+0000 7fdbcd255640 10 monclient(hunting): authenticate will time out at 1143.362205s 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc759e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdbc80a9d70 0x7fdbc814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46456/0 (socket says 172.21.15.45:46456) 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc759e640 1 -- 172.21.15.45:0/1347607685 learned_addr learned my addr 172.21.15.45:0/1347607685 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057227214 0 0) 0x7fdbc8145eb0 con 0x7fdbc80a9d70 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.125+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb0003520 con 0x7fdbc80a9d70 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503385459 0 0) 0x7fdbb0003520 con 0x7fdbc80a9d70 2024-03-29T11:38:07.129 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 >> v1:172.21.15.190:6789/0 conn(0x7fdbc81457f0 legacy=0x7fdbc813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 >> v1:172.21.15.45:6789/0 conn(0x7fdbc813f650 legacy=0x7fdbc80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8146340 con 0x7fdbc80a9d70 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdbc0001a10 con 0x7fdbc80a9d70 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: dump: 2024-03-29T11:38:07.130 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbcd255640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbcd255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbc0001be0 con 0x7fdbc80a9d70 2024-03-29T11:38:07.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 1 -- 172.21.15.45:0/1347607685 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdbc0002030 con 0x7fdbc80a9d70 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbc5d9b640 10 monclient: dump: 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:07.132 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbcd255640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.126+0000 7fdbcd255640 4 set_mon_vals no callback set 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 10 monclient: shutdown 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 1 -- 172.21.15.45:0/1347607685 >> v1:172.21.15.45:6790/0 conn(0x7fdbc80a9d70 legacy=0x7fdbc814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 1 -- 172.21.15.45:0/1347607685 shutdown_connections 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 1 -- 172.21.15.45:0/1347607685 >> 172.21.15.45:0/1347607685 conn(0x7fdbc8066a00 msgr2=0x7fdbc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 1 -- 172.21.15.45:0/1347607685 shutdown_connections 2024-03-29T11:38:07.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.127+0000 7fdbcd255640 1 -- 172.21.15.45:0/1347607685 wait complete. 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.128+0000 7fdbcd255640 10 monclient: build_initial_monmap 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.128+0000 7fdbcd255640 10 monclient: monmap: 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:07.141 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 librados: starting msgr at 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 librados: starting objecter 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 Processor -- start 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 -- start start 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 librados: setting wanted keys 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 1 librados: calling monclient init 2024-03-29T11:38:07.142 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.129+0000 7fdbcd255640 10 monclient: init 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc80a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc81457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc813e740 con 0x7fdbc813f650 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc8141450 con 0x7fdbc80a9d70 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.130+0000 7fdbcd255640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc812e180 con 0x7fdbc81457f0 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.131+0000 7fdbcd255640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.131+0000 7fdbcd255640 10 monclient(hunting): authenticate will time out at 1143.370207s 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.132+0000 7fdbc759e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fdbc81457f0 0x7fdbc8130610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47114/0 (socket says 172.21.15.45:47114) 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.132+0000 7fdbc759e640 1 -- 172.21.15.45:0/831772569 learned_addr learned my addr 172.21.15.45:0/831772569 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.132+0000 7fdbaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc81457f0 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.132+0000 7fdbaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.143 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.134+0000 7fdbaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc813f650 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.134+0000 7fdbaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.134+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47436854 0 0) 0x7fdbc813e740 con 0x7fdbc813f650 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.134+0000 7fdbaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.134+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb4004eb0 con 0x7fdbc813f650 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248761779 0 0) 0x7fdbc812e180 con 0x7fdbc81457f0 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc813e740 con 0x7fdbc81457f0 2024-03-29T11:38:07.144 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc80a9d70 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.135+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420860951 0 0) 0x7fdbc8141450 con 0x7fdbc80a9d70 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc812e180 con 0x7fdbc80a9d70 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703665789 0 0) 0x7fdbb4004eb0 con 0x7fdbc813f650 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc8141450 con 0x7fdbc813f650 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdbbc0017f0 con 0x7fdbc813f650 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:07.145 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.136+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092225695 0 0) 0x7fdbc812e180 con 0x7fdbc80a9d70 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb4004eb0 con 0x7fdbc80a9d70 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3258330572 0 0) 0x7fdbc8141450 con 0x7fdbc813f650 2024-03-29T11:38:07.146 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 >> v1:172.21.15.45:6790/0 conn(0x7fdbc81457f0 legacy=0x7fdbc8130610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 >> v1:172.21.15.190:6789/0 conn(0x7fdbc80a9d70 legacy=0x7fdbc81358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 monclient: found mon.a 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8142c40 con 0x7fdbc813f650 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 10 monclient: _renew_subs 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc813fda0 con 0x7fdbc813f650 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:07.147 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 10 monclient: _renew_subs 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.137+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc8130f00 con 0x7fdbc813f650 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbcd255640 1 librados: init done 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbbc001950 con 0x7fdbc813f650 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbc559a640 4 set_mon_vals no callback set 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fdbbc003790 con 0x7fdbc813f650 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.138+0000 7fdbaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.139+0000 7fdbaf7fe640 10 monclient: dump: 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.148 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.140+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fdbbc0038f0 con 0x7fdbc813f650 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.140+0000 7fdbcd255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.140+0000 7fdbcd255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.140+0000 7fdbcd255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:07.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.140+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba8005080 con 0x7fdbc813f650 2024-03-29T11:38:07.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.141+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fdbbc083710 con 0x7fdbc813f650 2024-03-29T11:38:07.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.141+0000 7fdbaf7fe640 10 client.4437.objecter ms_dispatch 0x7fdbc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:07.150 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.141+0000 7fdbaf7fe640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.141+0000 7fdbaf7fe640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.141+0000 7fdbc6d9d640 10 monclient: get_auth_request con 0x7fdbb406dcc0 auth_method 0 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.142+0000 7fdbaf7fe640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.142+0000 7fdbaf7fe640 10 client.4437.objecter ms_handle_connect 0x7fdbb406dcc0 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.149+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbbc04eff0 con 0x7fdbc813f650 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.149+0000 7fdbaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:07.151 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.149+0000 7fdbaf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:07.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.483+0000 7fdbcd255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.484+0000 7fdbcd255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.484+0000 7fdbcd255640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:07.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.484+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba8005330 con 0x7fdbc813f650 2024-03-29T11:38:07.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.485+0000 7fdbaf7fe640 1 -- 172.21.15.45:0/831772569 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fdbbc050f20 con 0x7fdbc813f650 2024-03-29T11:38:07.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.485+0000 7fdbaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.485+0000 7fdbaf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:07.486 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:07.486 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.489+0000 7fdbcd255640 10 librados: watch_flush enter 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.489+0000 7fdbcd255640 10 librados: watch_flush exit 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.489+0000 7fdbcd255640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.489+0000 7fdbcd255640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.490+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 >> v1:172.21.15.190:6816/1411598462 conn(0x7fdbb406dcc0 legacy=0x7fdbb4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.490+0000 7fdbcd255640 10 monclient: shutdown 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.490+0000 7fdbcd255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:07.490 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.490+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 >> v1:172.21.15.45:6789/0 conn(0x7fdbc813f650 legacy=0x7fdbc812da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.491+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 shutdown_connections 2024-03-29T11:38:07.491 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.491+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 >> 172.21.15.45:0/831772569 conn(0x7fdbc8066a00 msgr2=0x7fdbc80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:07.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.494+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 shutdown_connections 2024-03-29T11:38:07.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.494+0000 7fdbcd255640 1 -- 172.21.15.45:0/831772569 wait complete. 2024-03-29T11:38:07.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:07.494+0000 7fdbcd255640 1 librados: shutdown 2024-03-29T11:38:07.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:14.319 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:14.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.467+0000 7fbe06577640 10 monclient: get_monmap_and_config 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.467+0000 7fbe06577640 10 monclient: build_initial_monmap 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.467+0000 7fbe06577640 10 monclient: monmap: 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:14.468443+0000 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:14.468443+0000 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:14.468 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.470+0000 7fbe06577640 1 Processor -- start 2024-03-29T11:38:14.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 1 -- start start 2024-03-29T11:38:14.475 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient: init 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-b con 0x7fbe001457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0013f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-a con 0x7fbe000a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.471+0000 7fbe06577640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000aaef0 con 0x7fbe000a9d70 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbe06577640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00145bd0 con 0x7fbe0013f650 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbe06577640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00145eb0 con 0x7fbe001457f0 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbe06577640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbe06577640 10 monclient(hunting): authenticate will time out at 1150.711135s 2024-03-29T11:38:14.478 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbdfffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbe001457f0 0x7fbe0013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60644/0 (socket says 172.21.15.45:60644) 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.472+0000 7fbdfffff640 1 -- 172.21.15.45:0/1771802080 learned_addr learned my addr 172.21.15.45:0/1771802080 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090332753 0 0) 0x7fbe00145eb0 con 0x7fbe001457f0 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde8003520 con 0x7fbe001457f0 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345636398 0 0) 0x7fbde8003520 con 0x7fbe001457f0 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.473+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 >> v1:172.21.15.190:6789/0 conn(0x7fbe0013f650 legacy=0x7fbe000b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 >> v1:172.21.15.45:6789/0 conn(0x7fbe000a9d70 legacy=0x7fbe0014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 1 monclient: found mon.noname-b 2024-03-29T11:38:14.479 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00146340 con 0x7fbe001457f0 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbe06577640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbe06577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbdf8001a10 con 0x7fbe001457f0 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbdfeffd640 10 monclient: dump: 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:14.480 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.475+0000 7fbe06577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf8001ca0 con 0x7fbe001457f0 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbe06577640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbe06577640 4 set_mon_vals no callback set 2024-03-29T11:38:14.481 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 1 -- 172.21.15.45:0/1771802080 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbdf8002030 con 0x7fbe001457f0 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.476+0000 7fbdfeffd640 10 monclient: dump: 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:14.482 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.477+0000 7fbe06577640 10 monclient: shutdown 2024-03-29T11:38:14.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.477+0000 7fbe06577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.483 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.477+0000 7fbe06577640 1 -- 172.21.15.45:0/1771802080 >> v1:172.21.15.45:6790/0 conn(0x7fbe001457f0 legacy=0x7fbe0013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.478+0000 7fbe06577640 1 -- 172.21.15.45:0/1771802080 shutdown_connections 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.478+0000 7fbe06577640 1 -- 172.21.15.45:0/1771802080 >> 172.21.15.45:0/1771802080 conn(0x7fbe00066a00 msgr2=0x7fbe00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.479+0000 7fbe06577640 1 -- 172.21.15.45:0/1771802080 shutdown_connections 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.479+0000 7fbe06577640 1 -- 172.21.15.45:0/1771802080 wait complete. 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 10 monclient: build_initial_monmap 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 10 monclient: monmap: 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:14.484 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 librados: starting msgr at 2024-03-29T11:38:14.485 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 librados: starting objecter 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 Processor -- start 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 -- start start 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 librados: setting wanted keys 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 1 librados: calling monclient init 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.480+0000 7fbe06577640 10 monclient: init 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-a con 0x7fbe000a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:14.486 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0013f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient(hunting): picked mon.noname-c con 0x7fbe001457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00142150 con 0x7fbe000a9d70 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0006ee10 con 0x7fbe0013f650 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00142d50 con 0x7fbe001457f0 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:14.492 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.481+0000 7fbe06577640 10 monclient(hunting): authenticate will time out at 1150.720991s 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.485+0000 7fbe04b14640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fbe001457f0 0x7fbe001deab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33326/0 (socket says 172.21.15.45:33326) 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.485+0000 7fbe04b14640 1 -- 172.21.15.45:0/504643162 learned_addr learned my addr 172.21.15.45:0/504643162 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.485+0000 7fbdfcff9640 10 client.?.objecter ms_handle_connect 0x7fbe001457f0 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.485+0000 7fbdfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.485+0000 7fbdfcff9640 10 client.?.objecter ms_handle_connect 0x7fbe000a9d70 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272475579 0 0) 0x7fbe00142d50 con 0x7fbe001457f0 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec004eb0 con 0x7fbe001457f0 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346416784 0 0) 0x7fbe00142150 con 0x7fbe000a9d70 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe00142d50 con 0x7fbe000a9d70 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 client.?.objecter ms_handle_connect 0x7fbe0013f650 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.486+0000 7fbdfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736216437 0 0) 0x7fbe0006ee10 con 0x7fbe0013f650 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe00142150 con 0x7fbe0013f650 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2177299410 0 0) 0x7fbdec004eb0 con 0x7fbe001457f0 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe0006ee10 con 0x7fbe001457f0 2024-03-29T11:38:14.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbdf00018e0 con 0x7fbe001457f0 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): dump: 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:14.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2978267396 0 0) 0x7fbe0006ee10 con 0x7fbe001457f0 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 >> v1:172.21.15.190:6789/0 conn(0x7fbe0013f650 legacy=0x7fbe00136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 >> v1:172.21.15.45:6789/0 conn(0x7fbe000a9d70 legacy=0x7fbe00142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.488+0000 7fbdfcff9640 1 monclient: found mon.c 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0006de90 con 0x7fbe001457f0 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 10 monclient: _renew_subs 2024-03-29T11:38:14.496 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0006e100 con 0x7fbe001457f0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 10 monclient: _renew_subs 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe00137740 con 0x7fbe001457f0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbe06577640 1 librados: init done 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf0001a40 con 0x7fbe001457f0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fbdf0003850 con 0x7fbe001457f0 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfcff9640 10 monclient: dump: 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:14.497 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.489+0000 7fbdfe7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:14.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.491+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fbdf0003ab0 con 0x7fbe001457f0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.492+0000 7fbe06577640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.492+0000 7fbe06577640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.492+0000 7fbe06577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.492+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0005080 con 0x7fbe001457f0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfffff640 10 monclient: get_auth_request con 0x7fbdec06dbe0 auth_method 0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fbdf0084980 con 0x7fbe001457f0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfcff9640 10 client.4475.objecter ms_dispatch 0x7fbe00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfcff9640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfcff9640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.494+0000 7fbdfcff9640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:38:14.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.495+0000 7fbdfcff9640 10 client.4475.objecter ms_handle_connect 0x7fbdec06dbe0 2024-03-29T11:38:14.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.504+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbdf0050260 con 0x7fbe001457f0 2024-03-29T11:38:14.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.504+0000 7fbdfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.505 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.504+0000 7fbdfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:14.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.846+0000 7fbe06577640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.846+0000 7fbe06577640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.846+0000 7fbe06577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:14.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.846+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbde0005330 con 0x7fbe001457f0 2024-03-29T11:38:14.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.849+0000 7fbdfcff9640 1 -- 172.21.15.45:0/504643162 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fbdf0052190 con 0x7fbe001457f0 2024-03-29T11:38:14.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.849+0000 7fbdfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.848 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.849+0000 7fbdfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:14.849 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:14.849 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:14.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.852+0000 7fbe06577640 10 librados: watch_flush enter 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.853+0000 7fbe06577640 10 librados: watch_flush exit 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.853+0000 7fbe06577640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.853+0000 7fbe06577640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.854+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 >> v1:172.21.15.190:6816/1411598462 conn(0x7fbdec06dbe0 legacy=0x7fbdec0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.854+0000 7fbe06577640 10 monclient: shutdown 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.854+0000 7fbe06577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.854+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 >> v1:172.21.15.45:6790/0 conn(0x7fbe001457f0 legacy=0x7fbe001deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.855+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 shutdown_connections 2024-03-29T11:38:14.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.855+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 >> 172.21.15.45:0/504643162 conn(0x7fbe00066a00 msgr2=0x7fbe0013a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.855+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 shutdown_connections 2024-03-29T11:38:14.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.856+0000 7fbe06577640 1 -- 172.21.15.45:0/504643162 wait complete. 2024-03-29T11:38:14.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:14.856+0000 7fbe06577640 1 librados: shutdown 2024-03-29T11:38:14.874 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:21.680 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:21.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.829+0000 7fe79d44a640 10 monclient: get_monmap_and_config 2024-03-29T11:38:21.829 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.829+0000 7fe79d44a640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.829+0000 7fe79d44a640 10 monclient: monmap: 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:21.830091+0000 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:21.830091+0000 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:21.830 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.831 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.831+0000 7fe79d44a640 1 Processor -- start 2024-03-29T11:38:21.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.832+0000 7fe79d44a640 1 -- start start 2024-03-29T11:38:21.832 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.832+0000 7fe79d44a640 10 monclient: init 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-c con 0x7fe7980b1f60 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-b con 0x7fe798071eb0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-a con 0x7fe798143d10 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79806ef80 con 0x7fe798143d10 2024-03-29T11:38:21.833 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79806f260 con 0x7fe7980b1f60 2024-03-29T11:38:21.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980b40c0 con 0x7fe798071eb0 2024-03-29T11:38:21.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.834 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.833+0000 7fe79d44a640 10 monclient(hunting): authenticate will time out at 1158.071359s 2024-03-29T11:38:21.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.835+0000 7fe7967fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fe798071eb0 0x7fe798146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53826/0 (socket says 172.21.15.45:53826) 2024-03-29T11:38:21.835 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.835+0000 7fe7967fc640 1 -- 172.21.15.45:0/2706900426 learned_addr learned my addr 172.21.15.45:0/2706900426 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:21.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679425972 0 0) 0x7fe7980b40c0 con 0x7fe798071eb0 2024-03-29T11:38:21.836 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:38:21.837 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.836+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003520 con 0x7fe798071eb0 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44216685 0 0) 0x7fe780003520 con 0x7fe798071eb0 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 >> v1:172.21.15.190:6789/0 conn(0x7fe7980b1f60 legacy=0x7fe798143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 >> v1:172.21.15.45:6789/0 conn(0x7fe798143d10 legacy=0x7fe798146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 monclient: found mon.noname-b 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798155d70 con 0x7fe798071eb0 2024-03-29T11:38:21.841 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe788001a10 con 0x7fe798071eb0 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: dump: 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.842 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe788001be0 con 0x7fe798071eb0 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 1 -- 172.21.15.45:0/2706900426 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe788002030 con 0x7fe798071eb0 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe795ffb640 10 monclient: dump: 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.843 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe79d44a640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe79d44a640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.839+0000 7fe79d44a640 4 set_mon_vals no callback set 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.840+0000 7fe79d44a640 10 monclient: shutdown 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.840+0000 7fe79d44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.840+0000 7fe79d44a640 1 -- 172.21.15.45:0/2706900426 >> v1:172.21.15.45:6790/0 conn(0x7fe798071eb0 legacy=0x7fe798146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.841+0000 7fe79d44a640 1 -- 172.21.15.45:0/2706900426 shutdown_connections 2024-03-29T11:38:21.844 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.841+0000 7fe79d44a640 1 -- 172.21.15.45:0/2706900426 >> 172.21.15.45:0/2706900426 conn(0x7fe798066a00 msgr2=0x7fe798066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.844+0000 7fe79d44a640 1 -- 172.21.15.45:0/2706900426 shutdown_connections 2024-03-29T11:38:21.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.844+0000 7fe79d44a640 1 -- 172.21.15.45:0/2706900426 wait complete. 2024-03-29T11:38:21.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.845+0000 7fe79d44a640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.845 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.845+0000 7fe79d44a640 10 monclient: monmap: 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.845+0000 7fe79d44a640 1 librados: starting msgr at 2024-03-29T11:38:21.846 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.845+0000 7fe79d44a640 1 librados: starting objecter 2024-03-29T11:38:21.847 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 1 Processor -- start 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 1 -- start start 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 1 librados: setting wanted keys 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 1 librados: calling monclient init 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 10 monclient: init 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.850 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.846+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-b con 0x7fe7980b1f60 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-a con 0x7fe798143d10 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 10 monclient(hunting): picked mon.noname-c con 0x7fe798142490 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798140100 con 0x7fe798143d10 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798071240 con 0x7fe7980b1f60 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79806f750 con 0x7fe798142490 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe79d44a640 10 monclient(hunting): authenticate will time out at 1158.087902s 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe7967fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe798143d10 0x7fe79813f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:43193/0 (socket says 172.21.15.45:43193) 2024-03-29T11:38:21.852 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.849+0000 7fe7967fc640 1 -- 172.21.15.45:0/2728993730 learned_addr learned my addr 172.21.15.45:0/2728993730 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter ms_handle_connect 0x7fe798143d10 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter ms_handle_connect 0x7fe798142490 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7980b1f60 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033140607 0 0) 0x7fe79806f750 con 0x7fe798142490 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe784004eb0 con 0x7fe798142490 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738785869 0 0) 0x7fe798140100 con 0x7fe798143d10 2024-03-29T11:38:21.853 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79806f750 con 0x7fe798143d10 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648953721 0 0) 0x7fe798071240 con 0x7fe7980b1f60 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:38:21.854 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.852+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe798140100 con 0x7fe7980b1f60 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573734267 0 0) 0x7fe784004eb0 con 0x7fe798142490 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe798071240 con 0x7fe798142490 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe78c001680 con 0x7fe798142490 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.858 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952640966 0 0) 0x7fe79806f750 con 0x7fe798143d10 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe784004eb0 con 0x7fe798143d10 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe788001ef0 con 0x7fe798143d10 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.859 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 838213346 0 0) 0x7fe798140100 con 0x7fe7980b1f60 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe79806f750 con 0x7fe7980b1f60 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe790001830 con 0x7fe7980b1f60 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:21.860 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3908156034 0 0) 0x7fe798071240 con 0x7fe798142490 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 >> v1:172.21.15.190:6789/0 conn(0x7fe7980b1f60 legacy=0x7fe798154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 >> v1:172.21.15.45:6789/0 conn(0x7fe798143d10 legacy=0x7fe79813f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 monclient: found mon.c 2024-03-29T11:38:21.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798155620 con 0x7fe798142490 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe77f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe79d44a640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe79d44a640 10 monclient: _renew_subs 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe79d44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe798143040 con 0x7fe798142490 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.853+0000 7fe79d44a640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.854+0000 7fe79d44a640 10 monclient: _renew_subs 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.854+0000 7fe79d44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.854+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe798071660 con 0x7fe798142490 2024-03-29T11:38:21.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.854+0000 7fe79d44a640 1 librados: init done 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.855+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe78c001b90 con 0x7fe798142490 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.855+0000 7fe77f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.855+0000 7fe7957fa640 4 set_mon_vals no callback set 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.856+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe78c003790 con 0x7fe798142490 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.856+0000 7fe77f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.856+0000 7fe77f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.856+0000 7fe77f7fe640 10 monclient: dump: 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:21.863 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.856+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe78c0039b0 con 0x7fe798142490 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fe78c083670 con 0x7fe798142490 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe77f7fe640 10 client.4481.objecter ms_dispatch 0x7fe798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe77f7fe640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe77f7fe640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe77f7fe640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe79d44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe79d44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe79d44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:21.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.857+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe778005080 con 0x7fe798142490 2024-03-29T11:38:21.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.859+0000 7fe796ffd640 10 monclient: get_auth_request con 0x7fe78406e0d0 auth_method 0 2024-03-29T11:38:21.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.861+0000 7fe77f7fe640 10 client.4481.objecter ms_handle_connect 0x7fe78406e0d0 2024-03-29T11:38:21.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.865+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe78c04efd0 con 0x7fe798142490 2024-03-29T11:38:21.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.865+0000 7fe77f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:21.865+0000 7fe77f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.196+0000 7fe79d44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.196+0000 7fe79d44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.196+0000 7fe79d44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.196+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe778005330 con 0x7fe798142490 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.197+0000 7fe77f7fe640 1 -- 172.21.15.45:0/2728993730 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fe78c050f00 con 0x7fe798142490 2024-03-29T11:38:22.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.197+0000 7fe77f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:22.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.197+0000 7fe77f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:22.197 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:22.197 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:22.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.200+0000 7fe79d44a640 10 librados: watch_flush enter 2024-03-29T11:38:22.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.200+0000 7fe79d44a640 10 librados: watch_flush exit 2024-03-29T11:38:22.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.200+0000 7fe79d44a640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:38:22.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.200+0000 7fe79d44a640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:38:22.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.201+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe78406e0d0 legacy=0x7fe784070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.201+0000 7fe79d44a640 10 monclient: shutdown 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.201+0000 7fe79d44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.201+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 >> v1:172.21.15.45:6790/0 conn(0x7fe798142490 legacy=0x7fe79813f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.202+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 shutdown_connections 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.202+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 >> 172.21.15.45:0/2728993730 conn(0x7fe798066a00 msgr2=0x7fe7980b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.202+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 shutdown_connections 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.202+0000 7fe79d44a640 1 -- 172.21.15.45:0/2728993730 wait complete. 2024-03-29T11:38:22.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:22.202+0000 7fe79d44a640 1 librados: shutdown 2024-03-29T11:38:22.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:29.026 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.184+0000 7f055e643640 10 monclient: get_monmap_and_config 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.184+0000 7f055e643640 10 monclient: build_initial_monmap 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.184+0000 7f055e643640 10 monclient: monmap: 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:29.185636+0000 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:29.185636+0000 2024-03-29T11:38:29.185 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.185+0000 7f055e643640 1 Processor -- start 2024-03-29T11:38:29.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.185+0000 7f055e643640 1 -- start start 2024-03-29T11:38:29.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.185+0000 7f055e643640 10 monclient: init 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f055e643640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f055e643640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f055e643640 10 monclient(hunting): picked mon.noname-b con 0x7f05581422b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f055e643640 10 monclient(hunting): picked mon.noname-c con 0x7f0558142690 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f0557fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f05581422b0 0x7f055813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45638/0 (socket says 172.21.15.45:45638) 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f05577fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0558142690 0x7f05580b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:44716/0 (socket says 172.21.15.45:44716) 2024-03-29T11:38:29.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f055e643640 10 monclient(hunting): picked mon.noname-a con 0x7f05580b17d0 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.186+0000 7f0557fff640 1 -- 172.21.15.45:0/1390773618 learned_addr learned my addr 172.21.15.45:0/1390773618 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0558142c80 con 0x7f05580b17d0 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580aae40 con 0x7f0558142690 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055813fbd0 con 0x7f05581422b0 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f055e643640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f055e643640 10 monclient(hunting): authenticate will time out at 1165.426683s 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041202232 0 0) 0x7f05580aae40 con 0x7f0558142690 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0540003520 con 0x7f0558142690 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035661045 0 0) 0x7f055813fbd0 con 0x7f05581422b0 2024-03-29T11:38:29.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.187+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05580aae40 con 0x7f05581422b0 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282000319 0 0) 0x7f0540003520 con 0x7f0558142690 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 >> v1:172.21.15.45:6790/0 conn(0x7f05581422b0 legacy=0x7f055813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 >> v1:172.21.15.45:6789/0 conn(0x7f05580b17d0 legacy=0x7f05580b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 monclient: found mon.noname-c 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055814e910 con 0x7f0558142690 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:29.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f054c001690 con 0x7f0558142690 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: dump: 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f055e643640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f055e643640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f054c0017f0 con 0x7f0558142690 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 1 -- 172.21.15.45:0/1390773618 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f054c001b80 con 0x7f0558142690 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f0556ffd640 10 monclient: dump: 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:29.193 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:29.194 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:29.194 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f055e643640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:29.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.188+0000 7f055e643640 4 set_mon_vals no callback set 2024-03-29T11:38:29.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 10 monclient: shutdown 2024-03-29T11:38:29.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:29.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 >> v1:172.21.15.190:6789/0 conn(0x7f0558142690 legacy=0x7f05580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 shutdown_connections 2024-03-29T11:38:29.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 >> 172.21.15.45:0/1390773618 conn(0x7f0558066a00 msgr2=0x7f0558066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 shutdown_connections 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.189+0000 7f055e643640 1 -- 172.21.15.45:0/1390773618 wait complete. 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.191+0000 7f055e643640 10 monclient: build_initial_monmap 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.191+0000 7f055e643640 10 monclient: monmap: 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.191+0000 7f055e643640 1 librados: starting msgr at 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.191+0000 7f055e643640 1 librados: starting objecter 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.191+0000 7f055e643640 1 Processor -- start 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.192+0000 7f055e643640 1 -- start start 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.192+0000 7f055e643640 1 librados: setting wanted keys 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.192+0000 7f055e643640 1 librados: calling monclient init 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.192+0000 7f055e643640 10 monclient: init 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.193+0000 7f055e643640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.193+0000 7f055e643640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.193+0000 7f055e643640 10 monclient(hunting): picked mon.noname-b con 0x7f05580b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.193+0000 7f055e643640 10 monclient(hunting): picked mon.noname-c con 0x7f05581422b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0557fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f05580b17d0 0x7f0558134390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:45236/0 (socket says 172.21.15.45:45236) 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055cbe0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0558142690 0x7f05581e2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56370/0 (socket says 172.21.15.45:56370) 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055cbe0640 1 -- 172.21.15.45:0/848604429 learned_addr learned my addr 172.21.15.45:0/848604429 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 10 monclient(hunting): picked mon.noname-a con 0x7f0558142690 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:29.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0558132a40 con 0x7f0558142690 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580ab950 con 0x7f05580b17d0 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580abc30 con 0x7f05581422b0 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f055e643640 10 monclient(hunting): authenticate will time out at 1165.433532s 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter ms_handle_connect 0x7f05580b17d0 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter ms_handle_connect 0x7f0558142690 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter ms_handle_connect 0x7f05581422b0 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.194+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094063779 0 0) 0x7f05580ab950 con 0x7f05580b17d0 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0544004eb0 con 0x7f05580b17d0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306044827 0 0) 0x7f05580abc30 con 0x7f05581422b0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05580ab950 con 0x7f05581422b0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4173765083 0 0) 0x7f0544004eb0 con 0x7f05580b17d0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05580abc30 con 0x7f05580b17d0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0548001930 con 0x7f05580b17d0 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:29.207 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 915509839 0 0) 0x7f05580abc30 con 0x7f05580b17d0 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 >> v1:172.21.15.45:6790/0 conn(0x7f05581422b0 legacy=0x7f0558131320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 >> v1:172.21.15.45:6789/0 conn(0x7f0558142690 legacy=0x7f05581e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 monclient: found mon.b 2024-03-29T11:38:29.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580abf10 con 0x7f05580b17d0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f0554ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f055e643640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f055e643640 10 monclient: _renew_subs 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f055e643640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0558134aa0 con 0x7f05580b17d0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.195+0000 7f055e643640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f055e643640 10 monclient: _renew_subs 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f055e643640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05580ac330 con 0x7f05580b17d0 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f055e643640 1 librados: init done 2024-03-29T11:38:29.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05480047e0 con 0x7f05580b17d0 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f0548004bf0 con 0x7f05580b17d0 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f0554ff9640 10 monclient: dump: 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:29.210 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.196+0000 7f05567fc640 4 set_mon_vals no callback set 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.197+0000 7f055e643640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.198+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f0548004d50 con 0x7f05580b17d0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f055e643640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f055e643640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0538005080 con 0x7f05580b17d0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f0548084a80 con 0x7f05580b17d0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 10 client.4426.objecter ms_dispatch 0x7f0558000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f05577fe640 10 monclient: get_auth_request con 0x7f054406de00 auth_method 0 2024-03-29T11:38:29.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.199+0000 7f0554ff9640 10 client.4426.objecter ms_handle_connect 0x7f054406de00 2024-03-29T11:38:29.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.206+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0548050360 con 0x7f05580b17d0 2024-03-29T11:38:29.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.206+0000 7f0554ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:29.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.206+0000 7f0554ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:29.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.579+0000 7f055e643640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.579+0000 7f055e643640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.579+0000 7f055e643640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:38:29.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.580+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0538005330 con 0x7f05580b17d0 2024-03-29T11:38:29.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.582+0000 7f0554ff9640 1 -- 172.21.15.45:0/848604429 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f0548052290 con 0x7f05580b17d0 2024-03-29T11:38:29.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.582+0000 7f0554ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.582+0000 7f0554ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:29.583 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:29.583 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.585+0000 7f055e643640 10 librados: watch_flush enter 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.585+0000 7f055e643640 10 librados: watch_flush exit 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.585+0000 7f055e643640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.585+0000 7f055e643640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 >> v1:172.21.15.190:6816/1411598462 conn(0x7f054406de00 legacy=0x7f05440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 10 monclient: shutdown 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 >> v1:172.21.15.190:6789/0 conn(0x7f05580b17d0 legacy=0x7f0558134390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 shutdown_connections 2024-03-29T11:38:29.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 >> 172.21.15.45:0/848604429 conn(0x7f0558066a00 msgr2=0x7f055813d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:29.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 shutdown_connections 2024-03-29T11:38:29.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 -- 172.21.15.45:0/848604429 wait complete. 2024-03-29T11:38:29.588 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:29.586+0000 7f055e643640 1 librados: shutdown 2024-03-29T11:38:29.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:36.410 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:36.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.568+0000 7f5fab767640 10 monclient: get_monmap_and_config 2024-03-29T11:38:36.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.568+0000 7f5fab767640 10 monclient: build_initial_monmap 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.568+0000 7f5fab767640 10 monclient: monmap: 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:36.569666+0000 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:36.569666+0000 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:36.570 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.569+0000 7f5fab767640 1 Processor -- start 2024-03-29T11:38:36.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.570+0000 7f5fab767640 1 -- start start 2024-03-29T11:38:36.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.570+0000 7f5fab767640 10 monclient: init 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa40b0770 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-a con 0x7f5fa414a700 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-c con 0x7f5fa40ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40aba70 con 0x7f5fa414a700 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40aecb0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40af150 con 0x7f5fa40b0770 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:36.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.571+0000 7f5fab767640 10 monclient(hunting): authenticate will time out at 1172.810477s 2024-03-29T11:38:36.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.573+0000 7f5fa8d02640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5fa414a700 0x7f5fa40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52844/0 (socket says 172.21.15.45:52844) 2024-03-29T11:38:36.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.573+0000 7f5fa8d02640 1 -- 172.21.15.45:0/615805234 learned_addr learned my addr 172.21.15.45:0/615805234 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986653479 0 0) 0x7f5fa40aba70 con 0x7f5fa414a700 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:38:36.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.574+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f90003520 con 0x7f5fa414a700 2024-03-29T11:38:36.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.576+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641313150 0 0) 0x7f5fa40aecb0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.576+0000 7f5f9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.576+0000 7f5f9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:38:36.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa40aba70 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254205631 0 0) 0x7f5fa40aba70 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 >> v1:172.21.15.45:6790/0 conn(0x7f5fa40b0770 legacy=0x7f5fa4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.580 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.577+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 >> v1:172.21.15.45:6789/0 conn(0x7f5fa414a700 legacy=0x7f5fa40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 1 monclient: found mon.noname-c 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa40ac9d0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: _finish_auth 0 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5f9c0019b0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5f9bfff640 10 monclient: dump: 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.581 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5fab767640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.578+0000 7f5fab767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f9c001b80 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:36.582 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5fab767640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5fab767640 4 set_mon_vals no callback set 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 1 -- 172.21.15.45:0/615805234 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5f9c001f10 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.579+0000 7f5f9bfff640 10 monclient: dump: 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:36.583 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.580+0000 7f5fab767640 10 monclient: shutdown 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.580+0000 7f5fab767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.580+0000 7f5fab767640 1 -- 172.21.15.45:0/615805234 >> v1:172.21.15.190:6789/0 conn(0x7f5fa40ac5a0 legacy=0x7f5fa4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.580+0000 7f5fab767640 1 -- 172.21.15.45:0/615805234 shutdown_connections 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.580+0000 7f5fab767640 1 -- 172.21.15.45:0/615805234 >> 172.21.15.45:0/615805234 conn(0x7f5fa406b660 msgr2=0x7f5fa406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.581+0000 7f5fab767640 1 -- 172.21.15.45:0/615805234 shutdown_connections 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.581+0000 7f5fab767640 1 -- 172.21.15.45:0/615805234 wait complete. 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.582+0000 7f5fab767640 10 monclient: build_initial_monmap 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.582+0000 7f5fab767640 10 monclient: monmap: 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:36.585 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 librados: starting msgr at 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 librados: starting objecter 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 Processor -- start 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 -- start start 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 librados: setting wanted keys 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 1 librados: calling monclient init 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.583+0000 7f5fab767640 10 monclient: init 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:36.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa40ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-a con 0x7f5fa40b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient(hunting): picked mon.noname-c con 0x7f5fa414a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa41df900 con 0x7f5fa40b0770 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa4144db0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa41288c0 con 0x7f5fa414a700 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:36.587 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.584+0000 7f5fab767640 10 monclient(hunting): authenticate will time out at 1172.823241s 2024-03-29T11:38:36.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.588+0000 7f5fa8d02640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f5fa40b0770 0x7f5fa4143f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55874/0 (socket says 172.21.15.45:55874) 2024-03-29T11:38:36.589 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.588+0000 7f5fa8d02640 1 -- 172.21.15.45:0/908575942 learned_addr learned my addr 172.21.15.45:0/908575942 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 client.?.objecter ms_handle_connect 0x7f5fa40b0770 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875503703 0 0) 0x7f5fa41df900 con 0x7f5fa40b0770 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.591 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88004eb0 con 0x7f5fa40b0770 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 client.?.objecter ms_handle_connect 0x7f5fa40ac5a0 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.590+0000 7f5f99ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 client.?.objecter ms_handle_connect 0x7f5fa414a700 2024-03-29T11:38:36.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:36.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961543049 0 0) 0x7f5f88004eb0 con 0x7f5fa40b0770 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fa41df900 con 0x7f5fa40b0770 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5fa0001830 con 0x7f5fa40b0770 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:36.595 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.591+0000 7f5f99ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722382973 0 0) 0x7f5fa4144db0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88004eb0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 149013074 0 0) 0x7f5f88004eb0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f88005940 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5f940018b0 con 0x7f5fa40ac5a0 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:36.596 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3538419097 0 0) 0x7f5fa41df900 con 0x7f5fa40b0770 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 >> v1:172.21.15.45:6790/0 conn(0x7f5fa414a700 legacy=0x7f5fa41446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 >> v1:172.21.15.190:6789/0 conn(0x7f5fa40ac5a0 legacy=0x7f5fa4126260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 monclient: found mon.a 2024-03-29T11:38:36.597 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa41dfd90 con 0x7f5fa40b0770 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.592+0000 7f5f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 10 monclient: _renew_subs 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa4145b00 con 0x7f5fa40b0770 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 10 monclient: _renew_subs 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa4126b50 con 0x7f5fa40b0770 2024-03-29T11:38:36.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.593+0000 7f5fab767640 1 librados: init done 2024-03-29T11:38:36.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fa0001d20 con 0x7f5fa40b0770 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5fa00049c0 con 0x7f5fa40b0770 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f99ffb640 10 monclient: dump: 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:36.600 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.594+0000 7f5f9affd640 4 set_mon_vals no callback set 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.595+0000 7f5fab767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.595+0000 7f5fab767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.595+0000 7f5fab767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.595+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84005080 con 0x7f5fa40b0770 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.597+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f5fa0004b20 con 0x7f5fa40b0770 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f5fa00849e0 con 0x7f5fa40b0770 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5f99ffb640 10 client.4455.objecter ms_dispatch 0x7f5fa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5f99ffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5f99ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5fa9503640 10 monclient: get_auth_request con 0x7f5f8806e110 auth_method 0 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.598+0000 7f5f99ffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T11:38:36.601 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.599+0000 7f5f99ffb640 10 client.4455.objecter ms_handle_connect 0x7f5f8806e110 2024-03-29T11:38:36.608 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.608+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fa00500f0 con 0x7f5fa40b0770 2024-03-29T11:38:36.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.608+0000 7f5f99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.609 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.608+0000 7f5f99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:36.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.959+0000 7f5fab767640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.959+0000 7f5fab767640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.959+0000 7f5fab767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:36.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.959+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f84005330 con 0x7f5fa40b0770 2024-03-29T11:38:36.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.963+0000 7f5f99ffb640 1 -- 172.21.15.45:0/908575942 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f5fa00521f0 con 0x7f5fa40b0770 2024-03-29T11:38:36.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.963+0000 7f5f99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.963+0000 7f5f99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:36.965 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:36.965 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:36.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.967+0000 7f5fab767640 10 librados: watch_flush enter 2024-03-29T11:38:36.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.967+0000 7f5fab767640 10 librados: watch_flush exit 2024-03-29T11:38:36.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T11:38:36.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 10 client.4455.objecter successfully canceled tick 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 >> v1:172.21.15.190:6816/1411598462 conn(0x7f5f8806e110 legacy=0x7f5f880705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 10 monclient: shutdown 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 >> v1:172.21.15.45:6789/0 conn(0x7f5fa40b0770 legacy=0x7f5fa4143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 shutdown_connections 2024-03-29T11:38:36.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.968+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 >> 172.21.15.45:0/908575942 conn(0x7f5fa406b660 msgr2=0x7f5fa40ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.969+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 shutdown_connections 2024-03-29T11:38:36.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.969+0000 7f5fab767640 1 -- 172.21.15.45:0/908575942 wait complete. 2024-03-29T11:38:36.970 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:36.969+0000 7f5fab767640 1 librados: shutdown 2024-03-29T11:38:36.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:43.793 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.942+0000 7f113c5e2640 10 monclient: get_monmap_and_config 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.942+0000 7f113c5e2640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.942+0000 7f113c5e2640 10 monclient: monmap: 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:43.943671+0000 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:43.943671+0000 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:43.944 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:43.945 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:43.945 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.944+0000 7f113c5e2640 1 Processor -- start 2024-03-29T11:38:43.947 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.944+0000 7f113c5e2640 1 -- start start 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.944+0000 7f113c5e2640 10 monclient: init 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f11340a9930 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f1134149dc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f113414ff40 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.945+0000 7f113c5e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340aae60 con 0x7f1134149dc0 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113c5e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340b4310 con 0x7f113414ff40 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113c5e2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340afdd0 con 0x7f11340a9930 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113c5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113c5e2640 10 monclient(hunting): authenticate will time out at 1180.185129s 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113a37e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f11340a9930 0x7f1134143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35914/0 (socket says 172.21.15.45:35914) 2024-03-29T11:38:43.948 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.946+0000 7f113a37e640 1 -- 172.21.15.45:0/3400153948 learned_addr learned my addr 172.21.15.45:0/3400153948 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:43.949 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431509562 0 0) 0x7f11340afdd0 con 0x7f11340a9930 2024-03-29T11:38:43.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:38:43.950 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120003520 con 0x7f11340a9930 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.949+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408502878 0 0) 0x7f1120003520 con 0x7f11340a9930 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 >> v1:172.21.15.190:6789/0 conn(0x7f113414ff40 legacy=0x7f1134152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 >> v1:172.21.15.45:6789/0 conn(0x7f1134149dc0 legacy=0x7f113414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 1 monclient: found mon.noname-b 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113414a950 con 0x7f11340a9930 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.953 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f112c001a10 con 0x7f11340a9930 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113937c640 10 monclient: dump: 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:43.954 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:43.955 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113c5e2640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:38:43.955 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.950+0000 7f113c5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:43.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f112c001b70 con 0x7f11340a9930 2024-03-29T11:38:43.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 1 -- 172.21.15.45:0/3400153948 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f112c001f80 con 0x7f11340a9930 2024-03-29T11:38:43.956 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113937c640 10 monclient: dump: 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113c5e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:43.957 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.951+0000 7f113c5e2640 4 set_mon_vals no callback set 2024-03-29T11:38:43.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.952+0000 7f113c5e2640 10 monclient: shutdown 2024-03-29T11:38:43.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.952+0000 7f113c5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.952+0000 7f113c5e2640 1 -- 172.21.15.45:0/3400153948 >> v1:172.21.15.45:6790/0 conn(0x7f11340a9930 legacy=0x7f1134143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.952+0000 7f113c5e2640 1 -- 172.21.15.45:0/3400153948 shutdown_connections 2024-03-29T11:38:43.958 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.952+0000 7f113c5e2640 1 -- 172.21.15.45:0/3400153948 >> 172.21.15.45:0/3400153948 conn(0x7f11340b35e0 msgr2=0x7f11340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.953+0000 7f113c5e2640 1 -- 172.21.15.45:0/3400153948 shutdown_connections 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.953+0000 7f113c5e2640 1 -- 172.21.15.45:0/3400153948 wait complete. 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.954+0000 7f113c5e2640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.954+0000 7f113c5e2640 10 monclient: monmap: 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:43.959 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 librados: starting msgr at 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 librados: starting objecter 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 Processor -- start 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 -- start start 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 librados: setting wanted keys 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 1 librados: calling monclient init 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 10 monclient: init 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.960 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.955+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f11340a9930 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1134149dc0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f113414ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113414aae0 con 0x7f113414ff40 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11341490b0 con 0x7f11340a9930 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1134131c80 con 0x7f1134149dc0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.956+0000 7f113c5e2640 10 monclient(hunting): authenticate will time out at 1180.195213s 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f1139b7d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1134149dc0 0x7f1134130510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36406/0 (socket says 172.21.15.45:36406) 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f1139b7d640 1 -- 172.21.15.45:0/743913411 learned_addr learned my addr 172.21.15.45:0/743913411 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 client.?.objecter ms_handle_connect 0x7f1134149dc0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710927469 0 0) 0x7f1134131c80 con 0x7f1134149dc0 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.961 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.957+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1118004eb0 con 0x7f1134149dc0 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.958+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338933836 0 0) 0x7f1118004eb0 con 0x7f1134149dc0 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1134131c80 con 0x7f1134149dc0 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1124001a10 con 0x7f1134149dc0 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:43.962 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 10 monclient(hunting): dump: 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3838759470 0 0) 0x7f1134131c80 con 0x7f1134149dc0 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 >> v1:172.21.15.190:6789/0 conn(0x7f11340a9930 legacy=0x7f113408b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.963 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.959+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 >> v1:172.21.15.45:6789/0 conn(0x7f113414ff40 legacy=0x7f11341269c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.960+0000 7f112affd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.960+0000 7f112affd640 1 monclient: found mon.c 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.960+0000 7f112affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f112affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134131f60 con 0x7f1134149dc0 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f112affd640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f112affd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.964 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f112affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 10 monclient: _renew_subs 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1134132240 con 0x7f1134149dc0 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 10 monclient: _renew_subs 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1134148d20 con 0x7f1134149dc0 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.961+0000 7f113c5e2640 1 librados: init done 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f1128ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1124001b70 con 0x7f1134149dc0 2024-03-29T11:38:43.967 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f1128ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f1128ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c005080 con 0x7f1134149dc0 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f112affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.962+0000 7f1138b7b640 4 set_mon_vals no callback set 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.963+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1124003a60 con 0x7f1134149dc0 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.963+0000 7f112affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.963+0000 7f112affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.963+0000 7f112affd640 10 monclient: dump: 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:43.968 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.964+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f1124003bc0 con 0x7f1134149dc0 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f1124084b10 con 0x7f1134149dc0 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f112affd640 10 client.4499.objecter ms_dispatch 0x7f1134000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f112affd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f112affd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f112affd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.965+0000 7f113a37e640 10 monclient: get_auth_request con 0x7f111806dc20 auth_method 0 2024-03-29T11:38:43.969 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.966+0000 7f112affd640 10 client.4499.objecter ms_handle_connect 0x7f111806dc20 2024-03-29T11:38:43.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.972+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11240503f0 con 0x7f1134149dc0 2024-03-29T11:38:43.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.972+0000 7f112affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.973 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:43.972+0000 7f112affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:44.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.319+0000 7f1128ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.319+0000 7f1128ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.319+0000 7f1128ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:44.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.319+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f111c005330 con 0x7f1134149dc0 2024-03-29T11:38:44.323 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:44.323 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:44.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.322+0000 7f112affd640 1 -- 172.21.15.45:0/743913411 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f1124052320 con 0x7f1134149dc0 2024-03-29T11:38:44.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.322+0000 7f112affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:44.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.322+0000 7f112affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 10 librados: watch_flush enter 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 10 librados: watch_flush exit 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 >> v1:172.21.15.190:6816/1411598462 conn(0x7f111806dc20 legacy=0x7f11180700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 10 monclient: shutdown 2024-03-29T11:38:44.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.326+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 >> v1:172.21.15.45:6790/0 conn(0x7f1134149dc0 legacy=0x7f1134130510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.328+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 shutdown_connections 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.328+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 >> 172.21.15.45:0/743913411 conn(0x7f11340b35e0 msgr2=0x7f11340adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.329+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 shutdown_connections 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.329+0000 7f1128ff9640 1 -- 172.21.15.45:0/743913411 wait complete. 2024-03-29T11:38:44.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:44.329+0000 7f1128ff9640 1 librados: shutdown 2024-03-29T11:38:44.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:51.154 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.301+0000 7fc8fe233640 10 monclient: get_monmap_and_config 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.301+0000 7fc8fe233640 10 monclient: build_initial_monmap 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.301+0000 7fc8fe233640 10 monclient: monmap: 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:51.303113+0000 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:51.303113+0000 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.303 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.303+0000 7fc8fe233640 1 Processor -- start 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.303+0000 7fc8fe233640 1 -- start start 2024-03-29T11:38:51.304 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.303+0000 7fc8fe233640 10 monclient: init 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f81457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f80a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f80aaef0 con 0x7fc8f81457f0 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f8145bd0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f8145eb0 con 0x7fc8f813f650 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8fe233640 10 monclient(hunting): authenticate will time out at 1187.543459s 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8f77fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc8f81457f0 0x7fc8f813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:50334/0 (socket says 172.21.15.45:50334) 2024-03-29T11:38:51.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.304+0000 7fc8f77fe640 1 -- 172.21.15.45:0/2582847567 learned_addr learned my addr 172.21.15.45:0/2582847567 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150952699 0 0) 0x7fc8f80aaef0 con 0x7fc8f81457f0 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:38:51.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e0003520 con 0x7fc8f81457f0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975552550 0 0) 0x7fc8e0003520 con 0x7fc8f81457f0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 >> v1:172.21.15.45:6790/0 conn(0x7fc8f813f650 legacy=0x7fc8f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 >> v1:172.21.15.190:6789/0 conn(0x7fc8f80a9d70 legacy=0x7fc8f814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 monclient: found mon.noname-a 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8146340 con 0x7fc8f81457f0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:51.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc8e8001690 con 0x7fc8f81457f0 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.305+0000 7fc8f67fc640 10 monclient: dump: 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:51.310 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8fe233640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8fe233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8e8001860 con 0x7fc8f81457f0 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 1 -- 172.21.15.45:0/2582847567 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc8e8001cb0 con 0x7fc8f81457f0 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8f67fc640 10 monclient: dump: 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:51.311 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8fe233640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.306+0000 7fc8fe233640 4 set_mon_vals no callback set 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 10 monclient: shutdown 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 1 -- 172.21.15.45:0/2582847567 >> v1:172.21.15.45:6789/0 conn(0x7fc8f81457f0 legacy=0x7fc8f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 1 -- 172.21.15.45:0/2582847567 shutdown_connections 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 1 -- 172.21.15.45:0/2582847567 >> 172.21.15.45:0/2582847567 conn(0x7fc8f8066a00 msgr2=0x7fc8f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 1 -- 172.21.15.45:0/2582847567 shutdown_connections 2024-03-29T11:38:51.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.307+0000 7fc8fe233640 1 -- 172.21.15.45:0/2582847567 wait complete. 2024-03-29T11:38:51.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.308+0000 7fc8fe233640 10 monclient: build_initial_monmap 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.308+0000 7fc8fe233640 10 monclient: monmap: 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 librados: starting msgr at 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 librados: starting objecter 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 Processor -- start 2024-03-29T11:38:51.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 -- start start 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 librados: setting wanted keys 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 1 librados: calling monclient init 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.309+0000 7fc8fe233640 10 monclient: init 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f80a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f81457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8f6ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc8f813f650 0x7fc8f813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35542/0 (socket says 172.21.15.45:35542) 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8f6ffd640 1 -- 172.21.15.45:0/3767664316 learned_addr learned my addr 172.21.15.45:0/3767664316 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f8142ba0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8dffff640 10 client.?.objecter ms_handle_connect 0x7fc8f813f650 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f806e920 con 0x7fc8f81457f0 2024-03-29T11:38:51.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8dffff640 10 client.?.objecter ms_handle_connect 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f806eb90 con 0x7fc8f813f650 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.310+0000 7fc8fe233640 10 monclient(hunting): authenticate will time out at 1187.550070s 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190829941 0 0) 0x7fc8f8142ba0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4004eb0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747719224 0 0) 0x7fc8e4004eb0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8f8142ba0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc8e8001b70 con 0x7fc8f80a9d70 2024-03-29T11:38:51.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): dump: 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.311+0000 7fc8dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:51.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1634322843 0 0) 0x7fc8f8142ba0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 >> v1:172.21.15.45:6790/0 conn(0x7fc8f813f650 legacy=0x7fc8f813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 >> v1:172.21.15.190:6789/0 conn(0x7fc8f81457f0 legacy=0x7fc8f81e6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 1 monclient: found mon.a 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8142680 con 0x7fc8f80a9d70 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 10 monclient: _finish_auth 0 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 10 monclient: _renew_subs 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f81376e0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 10 monclient: _renew_subs 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f81423d0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.312+0000 7fc8fe233640 1 librados: init done 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8e8004a90 con 0x7fc8f80a9d70 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc8e8004da0 con 0x7fc8f80a9d70 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8dffff640 10 monclient: dump: 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:51.320 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8fe233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8fe233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8fe233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8f5ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:51.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.315+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fc8e801ad70 con 0x7fc8f80a9d70 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.314+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8075f10 con 0x7fc8f80a9d70 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.316+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fc8e8084b70 con 0x7fc8f80a9d70 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8dffff640 10 client.4464.objecter ms_dispatch 0x7fc8f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8dffff640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8dffff640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8f6ffd640 10 monclient: get_auth_request con 0x7fc8e406dbd0 auth_method 0 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8dffff640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:38:51.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.317+0000 7fc8dffff640 10 client.4464.objecter ms_handle_connect 0x7fc8e406dbd0 2024-03-29T11:38:51.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.323+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8e8084e60 con 0x7fc8f80a9d70 2024-03-29T11:38:51.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.323+0000 7fc8dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:51.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.323+0000 7fc8dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:51.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.709+0000 7fc8fe233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.709+0000 7fc8fe233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.709+0000 7fc8fe233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:38:51.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.709+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8f8142810 con 0x7fc8f80a9d70 2024-03-29T11:38:51.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.711+0000 7fc8dffff640 1 -- 172.21.15.45:0/3767664316 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fc8e8050450 con 0x7fc8f80a9d70 2024-03-29T11:38:51.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.711+0000 7fc8dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.711+0000 7fc8dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:51.713 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:51.714 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 10 librados: watch_flush enter 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 10 librados: watch_flush exit 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 >> v1:172.21.15.190:6816/1411598462 conn(0x7fc8e406dbd0 legacy=0x7fc8e4070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 10 monclient: shutdown 2024-03-29T11:38:51.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 >> v1:172.21.15.45:6789/0 conn(0x7fc8f80a9d70 legacy=0x7fc8f81db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 shutdown_connections 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 >> 172.21.15.45:0/3767664316 conn(0x7fc8f8066a00 msgr2=0x7fc8f813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 shutdown_connections 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 -- 172.21.15.45:0/3767664316 wait complete. 2024-03-29T11:38:51.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:51.715+0000 7fc8fe233640 1 librados: shutdown 2024-03-29T11:38:51.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:58.537 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.670+0000 7f2937c7b640 10 monclient: get_monmap_and_config 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.670+0000 7f2937c7b640 10 monclient: build_initial_monmap 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.670+0000 7f2937c7b640 10 monclient: monmap: 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:38:58.671287+0000 2024-03-29T11:38:58.671 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:38:58.671287+0000 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:38:58.672 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.670+0000 7f2937c7b640 1 Processor -- start 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 1 -- start start 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient: init 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f2930071de0 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f29300721c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f2930145ae0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930145650 con 0x7f2930071de0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300ae260 con 0x7f29300721c0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29301420d0 con 0x7f2930145ae0 2024-03-29T11:38:58.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.671+0000 7f2937c7b640 10 monclient(hunting): authenticate will time out at 1194.910908s 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2936218640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2930145ae0 0x7f2930145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49676/0 (socket says 172.21.15.45:49676) 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2936218640 1 -- 172.21.15.45:0/3314153810 learned_addr learned my addr 172.21.15.45:0/3314153810 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253226525 0 0) 0x7f29301420d0 con 0x7f2930145ae0 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.672+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c003520 con 0x7f2930145ae0 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.673+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109169261 0 0) 0x7f291c003520 con 0x7f2930145ae0 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.673+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 >> v1:172.21.15.190:6789/0 conn(0x7f29300721c0 legacy=0x7f293013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:58.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.673+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 >> v1:172.21.15.45:6789/0 conn(0x7f2930071de0 legacy=0x7f29300aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: _finish_hunting 0 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 1 monclient: found mon.noname-b 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300b0590 con 0x7f2930145ae0 2024-03-29T11:38:58.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: _finish_auth 0 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: _check_auth_tickets 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f292c001690 con 0x7f2930145ae0 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: dump: 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:58.677 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2937c7b640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2937c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f292c0017f0 con 0x7f2930145ae0 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2937c7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2937c7b640 4 set_mon_vals no callback set 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 1 -- 172.21.15.45:0/3314153810 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f292c001c00 con 0x7f2930145ae0 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.674+0000 7f2934a15640 10 monclient: dump: 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:58.678 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 10 monclient: shutdown 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 1 -- 172.21.15.45:0/3314153810 >> v1:172.21.15.45:6790/0 conn(0x7f2930145ae0 legacy=0x7f2930145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 1 -- 172.21.15.45:0/3314153810 shutdown_connections 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 1 -- 172.21.15.45:0/3314153810 >> 172.21.15.45:0/3314153810 conn(0x7f293006b660 msgr2=0x7f293006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:58.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 1 -- 172.21.15.45:0/3314153810 shutdown_connections 2024-03-29T11:38:58.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.675+0000 7f2937c7b640 1 -- 172.21.15.45:0/3314153810 wait complete. 2024-03-29T11:38:58.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.676+0000 7f2937c7b640 10 monclient: build_initial_monmap 2024-03-29T11:38:58.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.676+0000 7f2937c7b640 10 monclient: monmap: 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.677+0000 7f2937c7b640 1 librados: starting msgr at 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.677+0000 7f2937c7b640 1 librados: starting objecter 2024-03-29T11:38:58.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.677+0000 7f2937c7b640 1 Processor -- start 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.677+0000 7f2937c7b640 1 -- start start 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.677+0000 7f2937c7b640 1 librados: setting wanted keys 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.678+0000 7f2937c7b640 1 librados: calling monclient init 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.678+0000 7f2937c7b640 10 monclient: init 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.678+0000 7f2937c7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.678+0000 7f2937c7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.678+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f2930071de0 addr v1:172.21.15.45:6790/0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2935a17640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2930071de0 0x7f2930131b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:49872/0 (socket says 172.21.15.45:49872) 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f2930145ae0 addr v1:172.21.15.45:6789/0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2935a17640 1 -- 172.21.15.45:0/1548779390 learned_addr learned my addr 172.21.15.45:0/1548779390 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f29301442e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930144ba0 con 0x7f2930145ae0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930141e00 con 0x7f29301442e0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930144dd0 con 0x7f2930071de0 2024-03-29T11:38:58.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f2937c7b640 10 monclient(hunting): authenticate will time out at 1194.919016s 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f29267fc640 10 client.?.objecter ms_handle_connect 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.679+0000 7f29267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263351009 0 0) 0x7f2930144dd0 con 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914004eb0 con 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164439048 0 0) 0x7f2914004eb0 con 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2930144dd0 con 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2928001b50 con 0x7f2930071de0 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): dump: 2024-03-29T11:38:58.684 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 845869957 0 0) 0x7f2930144dd0 con 0x7f2930071de0 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 >> v1:172.21.15.190:6789/0 conn(0x7f29301442e0 legacy=0x7f2930143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.680+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 >> v1:172.21.15.45:6789/0 conn(0x7f2930145ae0 legacy=0x7f29300ac800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:58.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:58.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 1 monclient: found mon.c 2024-03-29T11:38:58.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:58.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29301450b0 con 0x7f2930071de0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f29267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 10 monclient: _renew_subs 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930145390 con 0x7f2930071de0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 10 monclient: _renew_subs 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.681+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29301e31c0 con 0x7f2930071de0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2928001cb0 con 0x7f2930071de0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2928004dc0 con 0x7f2930071de0 2024-03-29T11:38:58.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f2927fff640 4 set_mon_vals no callback set 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f29267fc640 10 monclient: dump: 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:38:58.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.682+0000 7f2937c7b640 1 librados: init done 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.683+0000 7f2937c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.683+0000 7f2937c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.683+0000 7f2937c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.683+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930141f90 con 0x7f2930071de0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.684+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f292801ad90 con 0x7f2930071de0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2928084e00 con 0x7f2930071de0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 10 client.4508.objecter ms_dispatch 0x7f2930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f2935216640 10 monclient: get_auth_request con 0x7f291406dcc0 auth_method 0 2024-03-29T11:38:58.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.685+0000 7f29267fc640 10 client.4508.objecter ms_handle_connect 0x7f291406dcc0 2024-03-29T11:38:58.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.692+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2928050510 con 0x7f2930071de0 2024-03-29T11:38:58.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.692+0000 7f29267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:58.692+0000 7f29267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:59.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f2937c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f2937c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f2937c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:38:59.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29300b2e50 con 0x7f2930071de0 2024-03-29T11:38:59.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f29267fc640 1 -- 172.21.15.45:0/1548779390 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f2928052610 con 0x7f2930071de0 2024-03-29T11:38:59.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f29267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:59.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.014+0000 7f29267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:59.017 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:38:59.017 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:59.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 10 librados: watch_flush enter 2024-03-29T11:38:59.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 10 librados: watch_flush exit 2024-03-29T11:38:59.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 >> v1:172.21.15.190:6816/1411598462 conn(0x7f291406dcc0 legacy=0x7f2914070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 10 monclient: shutdown 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 >> v1:172.21.15.45:6790/0 conn(0x7f2930071de0 legacy=0x7f2930131b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 shutdown_connections 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 >> 172.21.15.45:0/1548779390 conn(0x7f293006b660 msgr2=0x7f29300b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 shutdown_connections 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 -- 172.21.15.45:0/1548779390 wait complete. 2024-03-29T11:38:59.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:38:59.018+0000 7f2937c7b640 1 librados: shutdown 2024-03-29T11:38:59.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:05.837 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:05.980 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.979+0000 7f9c60e33640 10 monclient: get_monmap_and_config 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.979+0000 7f9c60e33640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.979+0000 7f9c60e33640 10 monclient: monmap: 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:05.980619+0000 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:05.980619+0000 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:05.981 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:05.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.980+0000 7f9c60e33640 1 Processor -- start 2024-03-29T11:39:05.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.980+0000 7f9c60e33640 1 -- start start 2024-03-29T11:39:05.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.980+0000 7f9c60e33640 10 monclient: init 2024-03-29T11:39:05.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.983 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0b4790 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c0aea90 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c0b39f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c5a59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9c5c0b4790 0x7f9c5c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:42782/0 (socket says 172.21.15.45:42782) 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c5a59c640 1 -- 172.21.15.45:0/2176041600 learned_addr learned my addr 172.21.15.45:0/2176041600 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.981+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0b0c60 con 0x7f9c5c0aea90 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0af020 con 0x7f9c5c0b39f0 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c13fbd0 con 0x7f9c5c0b4790 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c60e33640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c60e33640 10 monclient(hunting): authenticate will time out at 1202.221358s 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343770572 0 0) 0x7f9c5c13fbd0 con 0x7f9c5c0b4790 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.984 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.985 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:39:05.985 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.982+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c44003520 con 0x7f9c5c0b4790 2024-03-29T11:39:05.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911434203 0 0) 0x7f9c44003520 con 0x7f9c5c0b4790 2024-03-29T11:39:05.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 >> v1:172.21.15.190:6789/0 conn(0x7f9c5c0b39f0 legacy=0x7f9c5c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 >> v1:172.21.15.45:6789/0 conn(0x7f9c5c0aea90 legacy=0x7f9c5c0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 monclient: found mon.noname-b 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c13fd60 con 0x7f9c5c0b4790 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c60e33640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c60e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.983+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9c4c001690 con 0x7f9c5c0b4790 2024-03-29T11:39:05.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 10 monclient: dump: 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c60e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:05.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c4c001860 con 0x7f9c5c0b4790 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c5959a640 1 -- 172.21.15.45:0/2176041600 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9c4c001cb0 con 0x7f9c5c0b4790 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c60e33640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.984+0000 7f9c60e33640 4 set_mon_vals no callback set 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c5959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c5959a640 10 monclient: dump: 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:05.991 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c60e33640 10 monclient: shutdown 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c60e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.985+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 >> v1:172.21.15.45:6790/0 conn(0x7f9c5c0b4790 legacy=0x7f9c5c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.986+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 shutdown_connections 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.986+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 >> 172.21.15.45:0/2176041600 conn(0x7f9c5c066a00 msgr2=0x7f9c5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.986+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 shutdown_connections 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.986+0000 7f9c60e33640 1 -- 172.21.15.45:0/2176041600 wait complete. 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.987+0000 7f9c60e33640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.988+0000 7f9c60e33640 10 monclient: monmap: 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:05.992 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:05.993 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.988+0000 7f9c60e33640 1 librados: starting msgr at 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.988+0000 7f9c60e33640 1 librados: starting objecter 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.988+0000 7f9c60e33640 1 Processor -- start 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.989+0000 7f9c60e33640 1 -- start start 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.989+0000 7f9c60e33640 1 librados: setting wanted keys 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.989+0000 7f9c60e33640 1 librados: calling monclient init 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.989+0000 7f9c60e33640 10 monclient: init 2024-03-29T11:39:06.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c0b39f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0b4790 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c5a59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9c5c0b39f0 0x7f9c5c12c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43348/0 (socket says 172.21.15.45:43348) 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c5a59c640 1 -- 172.21.15.45:0/4287635600 learned_addr learned my addr 172.21.15.45:0/4287635600 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c4affd640 10 client.?.objecter ms_handle_connect 0x7f9c5c0b39f0 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c141780 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.990+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c131920 con 0x7f9c5c141780 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c142f90 con 0x7f9c5c0b4790 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c142360 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c60e33640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:06.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c60e33640 10 monclient(hunting): authenticate will time out at 1202.230198s 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 client.?.objecter ms_handle_connect 0x7f9c5c0b4790 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881533519 0 0) 0x7f9c5c142360 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.991+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c3c004eb0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773255635 0 0) 0x7f9c3c004eb0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5c142360 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9c4c001c30 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:06.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 10 monclient(hunting): dump: 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1950770644 0 0) 0x7f9c5c142360 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 >> v1:172.21.15.190:6789/0 conn(0x7f9c5c0b4790 legacy=0x7f9c5c131d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.992+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 >> v1:172.21.15.45:6789/0 conn(0x7f9c5c141780 legacy=0x7f9c5c142880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 1 monclient: found mon.c 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c135570 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: _finish_auth 0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 10 monclient: _renew_subs 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c1e2f30 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 10 monclient: _renew_subs 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c1e32a0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c60e33640 1 librados: init done 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c4c004a90 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9c4c004da0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c4affd640 10 monclient: dump: 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:06.005 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.994+0000 7f9c58d99640 4 set_mon_vals no callback set 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.995+0000 7f9c60e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.995+0000 7f9c60e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.995+0000 7f9c60e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.995+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c40005080 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.996+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f9c4c01ad70 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f9c4c084bd0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c4affd640 10 client.4514.objecter ms_dispatch 0x7f9c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c4affd640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c4affd640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c4affd640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.997+0000 7f9c59d9b640 10 monclient: get_auth_request con 0x7f9c3c06dc30 auth_method 0 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:05.998+0000 7f9c4affd640 10 client.4514.objecter ms_handle_connect 0x7f9c3c06dc30 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.005+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c4c0504b0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.005+0000 7f9c4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:06.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.005+0000 7f9c4affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:06.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.332+0000 7f9c60e33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:06.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.332+0000 7f9c60e33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:06.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.332+0000 7f9c60e33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.332+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c40005330 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.333+0000 7f9c4affd640 1 -- 172.21.15.45:0/4287635600 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f9c4c0523e0 con 0x7f9c5c0b39f0 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.333+0000 7f9c4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.333+0000 7f9c4affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:06.335 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.336+0000 7f9c60e33640 10 librados: watch_flush enter 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 10 librados: watch_flush exit 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 >> v1:172.21.15.190:6816/1411598462 conn(0x7f9c3c06dc30 legacy=0x7f9c3c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 10 monclient: shutdown 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 >> v1:172.21.15.45:6790/0 conn(0x7f9c5c0b39f0 legacy=0x7f9c5c12c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 shutdown_connections 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 >> 172.21.15.45:0/4287635600 conn(0x7f9c5c066a00 msgr2=0x7f9c5c0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 shutdown_connections 2024-03-29T11:39:06.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 -- 172.21.15.45:0/4287635600 wait complete. 2024-03-29T11:39:06.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:06.337+0000 7f9c60e33640 1 librados: shutdown 2024-03-29T11:39:06.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:13.160 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.307+0000 7f90e7fcf640 10 monclient: get_monmap_and_config 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.307+0000 7f90e7fcf640 10 monclient: build_initial_monmap 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.307+0000 7f90e7fcf640 10 monclient: monmap: 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:13.308977+0000 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:13.308977+0000 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:13.309 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.310 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:13.310 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:13.310 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:13.310 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.308+0000 7f90e7fcf640 1 Processor -- start 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.308+0000 7f90e7fcf640 1 -- start start 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.308+0000 7f90e7fcf640 10 monclient: init 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f90e0070170 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e5d6b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f90e0070170 0x7f90e013d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60368/0 (socket says 172.21.15.45:60368) 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e5d6b640 1 -- 172.21.15.45:0/854789899 learned_addr learned my addr 172.21.15.45:0/854789899 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f90e00713f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f90e0070550 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e013dbd0 con 0x7f90e00713f0 2024-03-29T11:39:13.312 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e013ff70 con 0x7f90e0070550 2024-03-29T11:39:13.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e0140250 con 0x7f90e0070170 2024-03-29T11:39:13.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:13.313 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.310+0000 7f90e7fcf640 10 monclient(hunting): authenticate will time out at 1209.549685s 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592889343 0 0) 0x7f90e0140250 con 0x7f90e0070170 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient(hunting): _negotiate 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc003520 con 0x7f90e0070170 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340923189 0 0) 0x7f90cc003520 con 0x7f90e0070170 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 >> v1:172.21.15.190:6789/0 conn(0x7f90e0070550 legacy=0x7f90e014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 >> v1:172.21.15.45:6789/0 conn(0x7f90e00713f0 legacy=0x7f90e01406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:13.318 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient: _finish_hunting 0 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 monclient: found mon.noname-b 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e0140df0 con 0x7f90e0070170 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.314+0000 7f90e4d69640 10 monclient: _finish_auth 0 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 10 monclient: _check_auth_tickets 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90dc001690 con 0x7f90e0070170 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e7fcf640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e7fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:13.319 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 10 monclient: dump: 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90dc001920 con 0x7f90e0070170 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e7fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e4d69640 1 -- 172.21.15.45:0/854789899 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90dc001cb0 con 0x7f90e0070170 2024-03-29T11:39:13.320 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e7fcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.315+0000 7f90e7fcf640 4 set_mon_vals no callback set 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e4d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e4d69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e4d69640 10 monclient: dump: 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:13.321 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e7fcf640 10 monclient: shutdown 2024-03-29T11:39:13.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e7fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:13.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.316+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 >> v1:172.21.15.45:6790/0 conn(0x7f90e0070170 legacy=0x7f90e013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.318+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 shutdown_connections 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.318+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 >> 172.21.15.45:0/854789899 conn(0x7f90e0067610 msgr2=0x7f90e00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.319+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 shutdown_connections 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.320+0000 7f90e7fcf640 1 -- 172.21.15.45:0/854789899 wait complete. 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.321+0000 7f90e7fcf640 10 monclient: build_initial_monmap 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.321+0000 7f90e7fcf640 10 monclient: monmap: 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.321+0000 7f90e7fcf640 1 librados: starting msgr at 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.321+0000 7f90e7fcf640 1 librados: starting objecter 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.321+0000 7f90e7fcf640 1 Processor -- start 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.322+0000 7f90e7fcf640 1 -- start start 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.323+0000 7f90e7fcf640 1 librados: setting wanted keys 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.323+0000 7f90e7fcf640 1 librados: calling monclient init 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.323+0000 7f90e7fcf640 10 monclient: init 2024-03-29T11:39:13.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.323+0000 7f90e7fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:13.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.323+0000 7f90e7fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:13.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.324+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f90e0070170 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:13.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.324+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f90e0070550 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f90e00713f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e012a620 con 0x7f90e0070550 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e012a3a0 con 0x7f90e00713f0 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e0132160 con 0x7f90e0070170 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e5d6b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f90e0070170 0x7f90e012d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60742/0 (socket says 172.21.15.45:60742) 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e7fcf640 10 monclient(hunting): authenticate will time out at 1209.564801s 2024-03-29T11:39:13.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.325+0000 7f90e5d6b640 1 -- 172.21.15.45:0/1093762779 learned_addr learned my addr 172.21.15.45:0/1093762779 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 client.?.objecter ms_handle_connect 0x7f90e0070170 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994371603 0 0) 0x7f90e0132160 con 0x7f90e0070170 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.327+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c4004eb0 con 0x7f90e0070170 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 client.?.objecter ms_handle_connect 0x7f90e0070550 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239262887 0 0) 0x7f90e012a620 con 0x7f90e0070550 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:13.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e0132160 con 0x7f90e0070550 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391364075 0 0) 0x7f90c4004eb0 con 0x7f90e0070170 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e012a620 con 0x7f90e0070170 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90dc001610 con 0x7f90e0070170 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): dump: 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:13.332 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.328+0000 7f90d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:13.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685342087 0 0) 0x7f90e012a620 con 0x7f90e0070170 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 >> v1:172.21.15.190:6789/0 conn(0x7f90e00713f0 legacy=0x7f90e0129c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 >> v1:172.21.15.45:6789/0 conn(0x7f90e0070550 legacy=0x7f90e0129580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 monclient: found mon.c 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e0130ca0 con 0x7f90e0070170 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:13.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 10 monclient: _renew_subs 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e012e060 con 0x7f90e0070170 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90dc004bf0 con 0x7f90e0070170 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f90dc004f00 con 0x7f90e0070170 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90d67fc640 10 monclient: dump: 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.335 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:13.336 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:13.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:13.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 10 monclient: _renew_subs 2024-03-29T11:39:13.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.329+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e012e4f0 con 0x7f90e0070170 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.330+0000 7f90e7fcf640 1 librados: init done 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.330+0000 7f90d7fff640 4 set_mon_vals no callback set 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.332+0000 7f90e7fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.332+0000 7f90e7fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.332+0000 7f90e7fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.338 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.332+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e0075f10 con 0x7f90e0070170 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.333+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f90dc01aed0 con 0x7f90e0070170 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f90dc084be0 con 0x7f90e0070170 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90d67fc640 10 client.4520.objecter ms_dispatch 0x7f90e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90d67fc640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90d67fc640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90d67fc640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:39:13.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.334+0000 7f90e556a640 10 monclient: get_auth_request con 0x7f90c4072110 auth_method 0 2024-03-29T11:39:13.340 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.335+0000 7f90d67fc640 10 client.4520.objecter ms_handle_connect 0x7f90c4072110 2024-03-29T11:39:13.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.342+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90dc0504c0 con 0x7f90e0070170 2024-03-29T11:39:13.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.342+0000 7f90d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:13.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.342+0000 7f90d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:13.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.672+0000 7f90e7fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.672+0000 7f90e7fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.672+0000 7f90e7fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:13.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.672+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90e01391e0 con 0x7f90e0070170 2024-03-29T11:39:13.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.673+0000 7f90d67fc640 1 -- 172.21.15.45:0/1093762779 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f90dc0523f0 con 0x7f90e0070170 2024-03-29T11:39:13.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.673+0000 7f90d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.673+0000 7f90d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:13.675 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:13.675 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:13.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.677+0000 7f90e7fcf640 10 librados: watch_flush enter 2024-03-29T11:39:13.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 10 librados: watch_flush exit 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 >> v1:172.21.15.190:6816/1411598462 conn(0x7f90c4072110 legacy=0x7f90c40745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 10 monclient: shutdown 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.678+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 >> v1:172.21.15.45:6790/0 conn(0x7f90e0070170 legacy=0x7f90e012d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.679+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 shutdown_connections 2024-03-29T11:39:13.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.679+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 >> 172.21.15.45:0/1093762779 conn(0x7f90e0067610 msgr2=0x7f90e013d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:13.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.679+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 shutdown_connections 2024-03-29T11:39:13.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.679+0000 7f90e7fcf640 1 -- 172.21.15.45:0/1093762779 wait complete. 2024-03-29T11:39:13.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:13.680+0000 7f90e7fcf640 1 librados: shutdown 2024-03-29T11:39:13.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:20.505 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.655+0000 7ff88eb2b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.655+0000 7ff88eb2b640 10 monclient: build_initial_monmap 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.655+0000 7ff88eb2b640 10 monclient: monmap: 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:20.655898+0000 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:20.655898+0000 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:20.656 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:20.657 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:20.657 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.656+0000 7ff88eb2b640 1 Processor -- start 2024-03-29T11:39:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 1 -- start start 2024-03-29T11:39:20.657 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient: init 2024-03-29T11:39:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:20.660 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8881457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff88813f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff8880a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8880aaef0 con 0x7ff8880a9d70 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff888145bd0 con 0x7ff88813f650 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff888145eb0 con 0x7ff8881457f0 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.657+0000 7ff88eb2b640 10 monclient(hunting): authenticate will time out at 1216.896001s 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.660+0000 7ff88c8c7640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff8881457f0 0x7ff88813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51524/0 (socket says 172.21.15.45:51524) 2024-03-29T11:39:20.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.660+0000 7ff88c8c7640 1 -- 172.21.15.45:0/1147929245 learned_addr learned my addr 172.21.15.45:0/1147929245 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063623170 0 0) 0x7ff888145eb0 con 0x7ff8881457f0 2024-03-29T11:39:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003520 con 0x7ff8881457f0 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970067426 0 0) 0x7ff888145bd0 con 0x7ff88813f650 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888145eb0 con 0x7ff88813f650 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258953076 0 0) 0x7ff870003520 con 0x7ff8881457f0 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 >> v1:172.21.15.190:6789/0 conn(0x7ff88813f650 legacy=0x7ff8880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 >> v1:172.21.15.45:6789/0 conn(0x7ff8880a9d70 legacy=0x7ff88814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff888146340 con 0x7ff8881457f0 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff878001690 con 0x7ff8881457f0 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: dump: 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:20.664 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff88eb2b640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff88eb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8780017f0 con 0x7ff8881457f0 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 1 -- 172.21.15.45:0/1147929245 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff878001c00 con 0x7ff8881457f0 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff87f7fe640 10 monclient: dump: 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff88eb2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:20.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.661+0000 7ff88eb2b640 4 set_mon_vals no callback set 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.662+0000 7ff88eb2b640 10 monclient: shutdown 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.662+0000 7ff88eb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.662+0000 7ff88eb2b640 1 -- 172.21.15.45:0/1147929245 >> v1:172.21.15.45:6790/0 conn(0x7ff8881457f0 legacy=0x7ff88813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.663+0000 7ff88eb2b640 1 -- 172.21.15.45:0/1147929245 shutdown_connections 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.663+0000 7ff88eb2b640 1 -- 172.21.15.45:0/1147929245 >> 172.21.15.45:0/1147929245 conn(0x7ff888066a00 msgr2=0x7ff888066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.663+0000 7ff88eb2b640 1 -- 172.21.15.45:0/1147929245 shutdown_connections 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.663+0000 7ff88eb2b640 1 -- 172.21.15.45:0/1147929245 wait complete. 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.664+0000 7ff88eb2b640 10 monclient: build_initial_monmap 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.664+0000 7ff88eb2b640 10 monclient: monmap: 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:20.668 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.664+0000 7ff88eb2b640 1 librados: starting msgr at 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.664+0000 7ff88eb2b640 1 librados: starting objecter 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.664+0000 7ff88eb2b640 1 Processor -- start 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.665+0000 7ff88eb2b640 1 -- start start 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.665+0000 7ff88eb2b640 1 librados: setting wanted keys 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.665+0000 7ff88eb2b640 1 librados: calling monclient init 2024-03-29T11:39:20.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.665+0000 7ff88eb2b640 10 monclient: init 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.666+0000 7ff88eb2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.666+0000 7ff88eb2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.666+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff8880a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.670+0000 7ff88c8c7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ff8880a9d70 0x7ff8881e14a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:54891/0 (socket says 172.21.15.45:54891) 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.670+0000 7ff87ffff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff88813f650 0x7ff8881e75b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:51938/0 (socket says 172.21.15.45:51938) 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.670+0000 7ff87ffff640 1 -- 172.21.15.45:0/3251538479 learned_addr learned my addr 172.21.15.45:0/3251538479 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 client.?.objecter ms_handle_connect 0x7ff88813f650 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.670+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff88813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8881457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88813ba90 con 0x7ff8880a9d70 2024-03-29T11:39:20.671 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88813b5b0 con 0x7ff8881457f0 2024-03-29T11:39:20.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff888072080 con 0x7ff88813f650 2024-03-29T11:39:20.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:20.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff88eb2b640 10 monclient(hunting): authenticate will time out at 1216.909317s 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169469397 0 0) 0x7ff888072080 con 0x7ff88813f650 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff874004eb0 con 0x7ff88813f650 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 client.?.objecter ms_handle_connect 0x7ff8880a9d70 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.671+0000 7ff87d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678268704 0 0) 0x7ff88813ba90 con 0x7ff8880a9d70 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888072080 con 0x7ff8880a9d70 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4006085511 0 0) 0x7ff888072080 con 0x7ff8880a9d70 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff874005490 con 0x7ff8880a9d70 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff878001b40 con 0x7ff8880a9d70 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): dump: 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:20.674 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2076405714 0 0) 0x7ff874005490 con 0x7ff8880a9d70 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 >> v1:172.21.15.45:6790/0 conn(0x7ff88813f650 legacy=0x7ff8881e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.675 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.672+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 >> v1:172.21.15.190:6789/0 conn(0x7ff8881457f0 legacy=0x7ff8881e7cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 1 monclient: found mon.a 2024-03-29T11:39:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:20.677 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88806faa0 con 0x7ff8880a9d70 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.674+0000 7ff87d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff878001d60 con 0x7ff8880a9d70 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff878001b40 con 0x7ff8880a9d70 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87d7fa640 10 monclient: dump: 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.678 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.675+0000 7ff87e7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 10 monclient: _renew_subs 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8881e1bb0 con 0x7ff8880a9d70 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 10 monclient: _renew_subs 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff888138b00 con 0x7ff8880a9d70 2024-03-29T11:39:20.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.676+0000 7ff88eb2b640 1 librados: init done 2024-03-29T11:39:20.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.678+0000 7ff88eb2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.678+0000 7ff88eb2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.678+0000 7ff88eb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:20.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.678+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff888075f10 con 0x7ff8880a9d70 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.678+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7ff87801a900 con 0x7ff8880a9d70 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.679+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7ff878084750 con 0x7ff8880a9d70 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.679+0000 7ff87d7fa640 10 client.4470.objecter ms_dispatch 0x7ff888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.679+0000 7ff87d7fa640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.679+0000 7ff87d7fa640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:20.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.679+0000 7ff87d7fa640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T11:39:20.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.681+0000 7ff87ffff640 10 monclient: get_auth_request con 0x7ff87406e180 auth_method 0 2024-03-29T11:39:20.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.683+0000 7ff87d7fa640 10 client.4470.objecter ms_handle_connect 0x7ff87406e180 2024-03-29T11:39:20.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.686+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff878050030 con 0x7ff8880a9d70 2024-03-29T11:39:20.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.686+0000 7ff87d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:20.686+0000 7ff87d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:21.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.015+0000 7ff88eb2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.015+0000 7ff88eb2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.015+0000 7ff88eb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.015+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8880b4990 con 0x7ff8880a9d70 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.016+0000 7ff87d7fa640 1 -- 172.21.15.45:0/3251538479 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7ff878051f60 con 0x7ff8880a9d70 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.016+0000 7ff87d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:21.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.016+0000 7ff87d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:21.017 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:21.017 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 10 librados: watch_flush enter 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 10 librados: watch_flush exit 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 10 client.4470.objecter successfully canceled tick 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 >> v1:172.21.15.190:6816/1411598462 conn(0x7ff87406e180 legacy=0x7ff874070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 10 monclient: shutdown 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 >> v1:172.21.15.45:6789/0 conn(0x7ff8880a9d70 legacy=0x7ff8881e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 shutdown_connections 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.020+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 >> 172.21.15.45:0/3251538479 conn(0x7ff888066a00 msgr2=0x7ff8880b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:21.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.021+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 shutdown_connections 2024-03-29T11:39:21.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.021+0000 7ff88eb2b640 1 -- 172.21.15.45:0/3251538479 wait complete. 2024-03-29T11:39:21.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:21.021+0000 7ff88eb2b640 1 librados: shutdown 2024-03-29T11:39:21.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:27.846 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:27.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.993+0000 7faf79f80640 10 monclient: get_monmap_and_config 2024-03-29T11:39:27.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.993+0000 7faf79f80640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.993+0000 7faf79f80640 10 monclient: monmap: 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:27.993436+0000 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:27.993436+0000 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.994+0000 7faf79f80640 1 Processor -- start 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.994+0000 7faf79f80640 1 -- start start 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.994+0000 7faf79f80640 10 monclient: init 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.995+0000 7faf79f80640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.995+0000 7faf79f80640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.995+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-a con 0x7faf6c0389e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-c con 0x7faf6c038dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-b con 0x7faf6c030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:27.998 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0c54c0 con 0x7faf6c0389e0 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c036430 con 0x7faf6c038dc0 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c039840 con 0x7faf6c030660 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.997+0000 7faf79f80640 10 monclient(hunting): authenticate will time out at 1224.235966s 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf7977f640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7faf6c030660 0x7faf6c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34630/0 (socket says 172.21.15.45:34630) 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf78f7e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7faf6c0389e0 0x7faf6c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:34045/0 (socket says 172.21.15.45:34045) 2024-03-29T11:39:27.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf7977f640 1 -- 172.21.15.45:0/4240913382 learned_addr learned my addr 172.21.15.45:0/4240913382 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497890908 0 0) 0x7faf6c039840 con 0x7faf6c030660 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c003520 con 0x7faf6c030660 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656580308 0 0) 0x7faf6c0c54c0 con 0x7faf6c0389e0 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.998+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c039840 con 0x7faf6c0389e0 2024-03-29T11:39:28.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.999+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 264451154 0 0) 0x7faf5c003520 con 0x7faf6c030660 2024-03-29T11:39:28.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:27.999+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 >> v1:172.21.15.190:6789/0 conn(0x7faf6c038dc0 legacy=0x7faf6c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:28.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 >> v1:172.21.15.45:6789/0 conn(0x7faf6c0389e0 legacy=0x7faf6c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c035690 con 0x7faf6c030660 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7faf7404f730 con 0x7faf6c030660 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf737fe640 10 monclient: dump: 2024-03-29T11:39:28.002 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf79f80640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:39:28.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.001+0000 7faf79f80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf740008d0 con 0x7faf6c030660 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 1 -- 172.21.15.45:0/4240913382 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7faf740758b0 con 0x7faf6c030660 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf737fe640 10 monclient: dump: 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:28.004 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf79f80640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf79f80640 4 set_mon_vals no callback set 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf79f80640 10 monclient: shutdown 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf79f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:28.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.003+0000 7faf79f80640 1 -- 172.21.15.45:0/4240913382 >> v1:172.21.15.45:6790/0 conn(0x7faf6c030660 legacy=0x7faf6c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.004+0000 7faf79f80640 1 -- 172.21.15.45:0/4240913382 shutdown_connections 2024-03-29T11:39:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.004+0000 7faf79f80640 1 -- 172.21.15.45:0/4240913382 >> 172.21.15.45:0/4240913382 conn(0x7faf6c005c00 msgr2=0x7faf6c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.005+0000 7faf79f80640 1 -- 172.21.15.45:0/4240913382 shutdown_connections 2024-03-29T11:39:28.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.005+0000 7faf79f80640 1 -- 172.21.15.45:0/4240913382 wait complete. 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.006+0000 7faf79f80640 10 monclient: build_initial_monmap 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.006+0000 7faf79f80640 10 monclient: monmap: 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:28.009 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 librados: starting msgr at 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 librados: starting objecter 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 Processor -- start 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 -- start start 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 librados: setting wanted keys 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 1 librados: calling monclient init 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.007+0000 7faf79f80640 10 monclient: init 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.008+0000 7faf79f80640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.008+0000 7faf79f80640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:28.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.008+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-c con 0x7faf6c030660 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-b con 0x7faf6c0389e0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 10 monclient(hunting): picked mon.noname-a con 0x7faf6c166d30 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0c8360 con 0x7faf6c166d30 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0c7160 con 0x7faf6c0389e0 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0c72f0 con 0x7faf6c030660 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:28.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.009+0000 7faf79f80640 10 monclient(hunting): authenticate will time out at 1224.247617s 2024-03-29T11:39:28.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf73fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7faf6c0389e0 0x7faf6c166620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:36283/0 (socket says 172.21.15.45:36283) 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf73fff640 1 -- 172.21.15.45:0/2877787357 learned_addr learned my addr 172.21.15.45:0/2877787357 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter ms_handle_connect 0x7faf6c0389e0 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter ms_handle_connect 0x7faf6c166d30 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter ms_handle_connect 0x7faf6c030660 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.012+0000 7faf717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985724498 0 0) 0x7faf6c0c8360 con 0x7faf6c166d30 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf60004eb0 con 0x7faf6c166d30 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371942402 0 0) 0x7faf6c0c7160 con 0x7faf6c0389e0 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c0c8360 con 0x7faf6c0389e0 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159572359 0 0) 0x7faf6c0c72f0 con 0x7faf6c030660 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c0c7160 con 0x7faf6c030660 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 826538084 0 0) 0x7faf60004eb0 con 0x7faf6c166d30 2024-03-29T11:39:28.019 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf6c0c72f0 con 0x7faf6c166d30 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7faf7404f400 con 0x7faf6c166d30 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): dump: 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:28.020 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1299795357 0 0) 0x7faf6c0c7160 con 0x7faf6c030660 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf60004eb0 con 0x7faf6c030660 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7faf64001a70 con 0x7faf6c030660 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.013+0000 7faf717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 10 monclient(hunting): dump: 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.021 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161315102 0 0) 0x7faf6c0c8360 con 0x7faf6c0389e0 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf6c0c7160 con 0x7faf6c0389e0 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955230094 0 0) 0x7faf6c0c72f0 con 0x7faf6c166d30 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 >> v1:172.21.15.45:6790/0 conn(0x7faf6c030660 legacy=0x7faf6c0c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 >> v1:172.21.15.190:6789/0 conn(0x7faf6c0389e0 legacy=0x7faf6c166620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 monclient: found mon.a 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:28.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.014+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c02f490 con 0x7faf6c166d30 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf7404f730 con 0x7faf6c166d30 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7faf74067d90 con 0x7faf6c166d30 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf72ffd640 4 set_mon_vals no callback set 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf717fa640 10 monclient: dump: 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:28.023 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf79f80640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf79f80640 10 monclient: _renew_subs 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.015+0000 7faf79f80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c02f740 con 0x7faf6c166d30 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 10 monclient: _renew_subs 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c169de0 con 0x7faf6c166d30 2024-03-29T11:39:28.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.016+0000 7faf79f80640 1 librados: init done 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.018+0000 7faf79f80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.018+0000 7faf79f80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.018+0000 7faf79f80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.018+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c007b70 con 0x7faf6c166d30 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.018+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7faf74067ef0 con 0x7faf6c166d30 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7faf740ff340 con 0x7faf6c166d30 2024-03-29T11:39:28.025 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf717fa640 10 client.4476.objecter ms_dispatch 0x7faf74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf717fa640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf717fa640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf717fa640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:39:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.019+0000 7faf78f7e640 10 monclient: get_auth_request con 0x7faf6006dff0 auth_method 0 2024-03-29T11:39:28.026 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.020+0000 7faf717fa640 10 client.4476.objecter ms_handle_connect 0x7faf6006dff0 2024-03-29T11:39:28.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.029+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf740cac20 con 0x7faf6c166d30 2024-03-29T11:39:28.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.029+0000 7faf717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:28.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.029+0000 7faf717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.360+0000 7faf79f80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.360+0000 7faf79f80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:28.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.360+0000 7faf79f80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.360+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf6c0c8130 con 0x7faf6c166d30 2024-03-29T11:39:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.362+0000 7faf717fa640 1 -- 172.21.15.45:0/2877787357 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7faf740ccb50 con 0x7faf6c166d30 2024-03-29T11:39:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.362+0000 7faf717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:28.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.362+0000 7faf717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:28.365 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:28.365 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 10 librados: watch_flush enter 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 10 librados: watch_flush exit 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 >> v1:172.21.15.190:6816/1411598462 conn(0x7faf6006dff0 legacy=0x7faf600704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 10 monclient: shutdown 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 >> v1:172.21.15.45:6789/0 conn(0x7faf6c166d30 legacy=0x7faf6c1694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 shutdown_connections 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 >> 172.21.15.45:0/2877787357 conn(0x7faf6c005c00 msgr2=0x7faf6c03ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 shutdown_connections 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 -- 172.21.15.45:0/2877787357 wait complete. 2024-03-29T11:39:28.369 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:28.368+0000 7faf79f80640 1 librados: shutdown 2024-03-29T11:39:28.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:35.192 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:35.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.341+0000 7fcdb5c4d640 10 monclient: get_monmap_and_config 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.341+0000 7fcdb5c4d640 10 monclient: build_initial_monmap 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.341+0000 7fcdb5c4d640 10 monclient: monmap: 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:35.342658+0000 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:35.342658+0000 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.384 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:35.385 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:35.385 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:35.385 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.394 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.342+0000 7fcdb5c4d640 1 Processor -- start 2024-03-29T11:39:35.394 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.342+0000 7fcdb5c4d640 1 -- start start 2024-03-29T11:39:35.394 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.342+0000 7fcdb5c4d640 10 monclient: init 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb00b4710 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb00b3970 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb013f390 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb00af020 con 0x7fcdb00b4710 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb00b0910 con 0x7fcdb013f390 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb00b0bf0 con 0x7fcdb00b3970 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.343+0000 7fcdb5c4d640 10 monclient(hunting): authenticate will time out at 1231.582780s 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdaffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcdb013f390 0x7fcdb013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:52606/0 (socket says 172.21.15.45:52606) 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdaffff640 1 -- 172.21.15.45:0/3304832349 learned_addr learned my addr 172.21.15.45:0/3304832349 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869049535 0 0) 0x7fcdb00af020 con 0x7fcdb00b4710 2024-03-29T11:39:35.395 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.344+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98003520 con 0x7fcdb00b4710 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614778773 0 0) 0x7fcdb00b0bf0 con 0x7fcdb00b3970 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb00af020 con 0x7fcdb00b3970 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 60795730 0 0) 0x7fcd98003520 con 0x7fcdb00b4710 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 >> v1:172.21.15.45:6790/0 conn(0x7fcdb00b3970 legacy=0x7fcdb00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 >> v1:172.21.15.190:6789/0 conn(0x7fcdb013f390 legacy=0x7fcdb013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:35.396 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 monclient: found mon.noname-a 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb00aeb00 con 0x7fcdb00b4710 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcda4001690 con 0x7fcdb00b4710 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: dump: 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.397 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdb5c4d640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdb5c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcda40017f0 con 0x7fcdb00b4710 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 1 -- 172.21.15.45:0/3304832349 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcda4001c40 con 0x7fcdb00b4710 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdae7fc640 10 monclient: dump: 2024-03-29T11:39:35.398 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdb5c4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.345+0000 7fcdb5c4d640 4 set_mon_vals no callback set 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 10 monclient: shutdown 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:35.399 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3304832349 >> v1:172.21.15.45:6789/0 conn(0x7fcdb00b4710 legacy=0x7fcdb00b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3304832349 shutdown_connections 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3304832349 >> 172.21.15.45:0/3304832349 conn(0x7fcdb0066a00 msgr2=0x7fcdb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.346+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3304832349 shutdown_connections 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.347+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3304832349 wait complete. 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.347+0000 7fcdb5c4d640 10 monclient: build_initial_monmap 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 10 monclient: monmap: 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:35.400 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 librados: starting msgr at 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 librados: starting objecter 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 Processor -- start 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 -- start start 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 librados: setting wanted keys 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 1 librados: calling monclient init 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.348+0000 7fcdb5c4d640 10 monclient: init 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb00b3970 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb00b4710 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb013f390 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb012d5f0 con 0x7fcdb00b3970 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb012fad0 con 0x7fcdb00b4710 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb0140880 con 0x7fcdb013f390 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.349+0000 7fcdb5c4d640 10 monclient(hunting): authenticate will time out at 1231.588949s 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcdaeffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcdb00b4710 0x7fcdb012c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:56640/0 (socket says 172.21.15.45:56640) 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcdaeffd640 1 -- 172.21.15.45:0/3816139314 learned_addr learned my addr 172.21.15.45:0/3816139314 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter ms_handle_connect 0x7fcdb00b4710 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter ms_handle_connect 0x7fcdb013f390 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287719406 0 0) 0x7fcdb012fad0 con 0x7fcdb00b4710 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9c004eb0 con 0x7fcdb00b4710 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter ms_handle_connect 0x7fcdb00b3970 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.350+0000 7fcd97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653785237 0 0) 0x7fcdb0140880 con 0x7fcdb013f390 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb012fad0 con 0x7fcdb013f390 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881281731 0 0) 0x7fcdb012d5f0 con 0x7fcdb00b3970 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb0140880 con 0x7fcdb00b3970 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.351+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951555255 0 0) 0x7fcd9c004eb0 con 0x7fcdb00b4710 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdb012d5f0 con 0x7fcdb00b4710 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcda8001dd0 con 0x7fcdb00b4710 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): dump: 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931084720 0 0) 0x7fcdb0140880 con 0x7fcdb00b3970 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd9c004eb0 con 0x7fcdb00b3970 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcda4001c40 con 0x7fcdb00b3970 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient(hunting): dump: 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669266110 0 0) 0x7fcdb012d5f0 con 0x7fcdb00b4710 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 >> v1:172.21.15.45:6790/0 conn(0x7fcdb013f390 legacy=0x7fcdb012cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 >> v1:172.21.15.45:6789/0 conn(0x7fcdb00b3970 legacy=0x7fcdb0140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 monclient: found mon.b 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.352+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb012fd70 con 0x7fcdb00b4710 2024-03-29T11:39:35.406 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcd97fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcd97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcd97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 10 monclient: _renew_subs 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb013ff70 con 0x7fcdb00b4710 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 10 monclient: _renew_subs 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb01396b0 con 0x7fcdb00b4710 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.353+0000 7fcdb5c4d640 1 librados: init done 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcda8001fb0 con 0x7fcdb00b4710 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fcda8003de0 con 0x7fcdb00b4710 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:35.407 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcdadffb640 4 set_mon_vals no callback set 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.354+0000 7fcd97fff640 10 monclient: dump: 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.355+0000 7fcdb5c4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.355+0000 7fcdb5c4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.355+0000 7fcdb5c4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.355+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd90005080 con 0x7fcdb00b4710 2024-03-29T11:39:35.408 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.355+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fcda8004000 con 0x7fcdb00b4710 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fcda8084e40 con 0x7fcdb00b4710 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcd97fff640 10 client.4447.objecter ms_dispatch 0x7fcdb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcd97fff640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcd97fff640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcd97fff640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.356+0000 7fcdaf7fe640 10 monclient: get_auth_request con 0x7fcd9c06dd00 auth_method 0 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.357+0000 7fcd97fff640 10 client.4447.objecter ms_handle_connect 0x7fcd9c06dd00 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.365+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcda8050720 con 0x7fcdb00b4710 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.365+0000 7fcd97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:35.409 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.365+0000 7fcd97fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:35.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.703+0000 7fcdb5c4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.703+0000 7fcdb5c4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.703+0000 7fcdb5c4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:35.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.703+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd90005330 con 0x7fcdb00b4710 2024-03-29T11:39:35.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.705+0000 7fcd97fff640 1 -- 172.21.15.45:0/3816139314 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fcda8052650 con 0x7fcdb00b4710 2024-03-29T11:39:35.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.705+0000 7fcd97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.705+0000 7fcd97fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:35.707 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:35.707 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 10 librados: watch_flush enter 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 10 librados: watch_flush exit 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 >> v1:172.21.15.190:6816/1411598462 conn(0x7fcd9c06dd00 legacy=0x7fcd9c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 10 monclient: shutdown 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:35.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 >> v1:172.21.15.190:6789/0 conn(0x7fcdb00b4710 legacy=0x7fcdb012c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 shutdown_connections 2024-03-29T11:39:35.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 >> 172.21.15.45:0/3816139314 conn(0x7fcdb0066a00 msgr2=0x7fcdb00b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:35.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 shutdown_connections 2024-03-29T11:39:35.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 -- 172.21.15.45:0/3816139314 wait complete. 2024-03-29T11:39:35.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:35.708+0000 7fcdb5c4d640 1 librados: shutdown 2024-03-29T11:39:35.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:42.531 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:42.703 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.702+0000 7f64c7179640 10 monclient: get_monmap_and_config 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.702+0000 7f64c7179640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.702+0000 7f64c7179640 10 monclient: monmap: 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:42.703515+0000 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:42.703515+0000 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:42.704 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:42.705 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:42.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.706+0000 7f64c7179640 1 Processor -- start 2024-03-29T11:39:42.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.706+0000 7f64c7179640 1 -- start start 2024-03-29T11:39:42.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.706+0000 7f64c7179640 10 monclient: init 2024-03-29T11:39:42.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-a con 0x7f64c01422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-c con 0x7f64c0142690 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0142c80 con 0x7f64c01422b0 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00aae40 con 0x7f64c0142690 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c013fbd0 con 0x7f64c00b17d0 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.707+0000 7f64c7179640 10 monclient(hunting): authenticate will time out at 1238.946793s 2024-03-29T11:39:42.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.710+0000 7f64bffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f64c0142690 0x7f64c00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:46797/0 (socket says 172.21.15.45:46797) 2024-03-29T11:39:42.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.710+0000 7f64bffff640 1 -- 172.21.15.45:0/1445501579 learned_addr learned my addr 172.21.15.45:0/1445501579 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:42.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017772388 0 0) 0x7f64c00aae40 con 0x7f64c0142690 2024-03-29T11:39:42.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:39:42.713 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.711+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0003520 con 0x7f64c0142690 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154912783 0 0) 0x7f64c013fbd0 con 0x7f64c00b17d0 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c00aae40 con 0x7f64c00b17d0 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27797819 0 0) 0x7f64c00aae40 con 0x7f64c00b17d0 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 >> v1:172.21.15.190:6789/0 conn(0x7f64c0142690 legacy=0x7f64c00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.714 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.712+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 >> v1:172.21.15.45:6789/0 conn(0x7f64c01422b0 legacy=0x7f64c013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:42.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:42.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.716 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0151940 con 0x7f64c00b17d0 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b80019f0 con 0x7f64c00b17d0 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.714+0000 7f64bf7fe640 10 monclient: dump: 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.717 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64c7179640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:39:42.718 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64c7179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64bf7fe640 1 -- 172.21.15.45:0/1445501579 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b8001b50 con 0x7f64c00b17d0 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64c7179640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.715+0000 7f64c7179640 4 set_mon_vals no callback set 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.716+0000 7f64c7179640 10 monclient: shutdown 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.716+0000 7f64c7179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.719 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.716+0000 7f64c7179640 1 -- 172.21.15.45:0/1445501579 >> v1:172.21.15.45:6790/0 conn(0x7f64c00b17d0 legacy=0x7f64c00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.718+0000 7f64c7179640 1 -- 172.21.15.45:0/1445501579 shutdown_connections 2024-03-29T11:39:42.720 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.718+0000 7f64c7179640 1 -- 172.21.15.45:0/1445501579 >> 172.21.15.45:0/1445501579 conn(0x7f64c0066a00 msgr2=0x7f64c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.719+0000 7f64c7179640 1 -- 172.21.15.45:0/1445501579 shutdown_connections 2024-03-29T11:39:42.721 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.719+0000 7f64c7179640 1 -- 172.21.15.45:0/1445501579 wait complete. 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.720+0000 7f64c7179640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.720+0000 7f64c7179640 10 monclient: monmap: 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:42.722 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.720+0000 7f64c7179640 1 librados: starting msgr at 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 1 librados: starting objecter 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 1 Processor -- start 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 1 -- start start 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 1 librados: setting wanted keys 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 1 librados: calling monclient init 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.721+0000 7f64c7179640 10 monclient: init 2024-03-29T11:39:42.724 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.722+0000 7f64c7179640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.722+0000 7f64c7179640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-a con 0x7f64c01422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 10 monclient(hunting): picked mon.noname-c con 0x7f64c0142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0131bf0 con 0x7f64c01422b0 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c006eab0 con 0x7f64c00b17d0 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c006ed90 con 0x7f64c0142690 2024-03-29T11:39:42.725 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.726 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.723+0000 7f64c7179640 10 monclient(hunting): authenticate will time out at 1238.962808s 2024-03-29T11:39:42.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.725+0000 7f64c5716640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f64c0142690 0x7f64c0071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54170/0 (socket says 172.21.15.45:54170) 2024-03-29T11:39:42.727 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.725+0000 7f64c5716640 1 -- 172.21.15.45:0/1670035873 learned_addr learned my addr 172.21.15.45:0/1670035873 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:42.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.727+0000 7f64bd7fa640 10 client.?.objecter ms_handle_connect 0x7f64c0142690 2024-03-29T11:39:42.728 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.727+0000 7f64bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.727+0000 7f64bd7fa640 10 client.?.objecter ms_handle_connect 0x7f64c00b17d0 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.727+0000 7f64bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544436032 0 0) 0x7f64c006eab0 con 0x7f64c00b17d0 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4004eb0 con 0x7f64c00b17d0 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102244620 0 0) 0x7f64a4004eb0 con 0x7f64c00b17d0 2024-03-29T11:39:42.731 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.728+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c006eab0 con 0x7f64c00b17d0 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b40018a0 con 0x7f64c00b17d0 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:42.732 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:42.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:42.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3917867595 0 0) 0x7f64c006eab0 con 0x7f64c00b17d0 2024-03-29T11:39:42.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 >> v1:172.21.15.45:6790/0 conn(0x7f64c0142690 legacy=0x7f64c0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.733 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.729+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 >> v1:172.21.15.45:6789/0 conn(0x7f64c01422b0 legacy=0x7f64c0071460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 1 monclient: found mon.b 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c01deae0 con 0x7f64c00b17d0 2024-03-29T11:39:42.734 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 10 monclient: _renew_subs 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0132820 con 0x7f64c00b17d0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 10 monclient: _renew_subs 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c013bb60 con 0x7f64c00b17d0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64c7179640 1 librados: init done 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b4001a00 con 0x7f64c00b17d0 2024-03-29T11:39:42.735 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.731+0000 7f64beffd640 4 set_mon_vals no callback set 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.732+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f64b4003880 con 0x7f64c00b17d0 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.732+0000 7f64bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.732+0000 7f64bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.732+0000 7f64bd7fa640 10 monclient: dump: 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:42.736 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:42.737 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:42.737 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:42.737 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.736+0000 7f64c7179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.736+0000 7f64c7179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.736+0000 7f64c7179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:42.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.736+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0005080 con 0x7f64c00b17d0 2024-03-29T11:39:42.738 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f64b4019850 con 0x7f64c00b17d0 2024-03-29T11:39:42.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f64b4083820 con 0x7f64c00b17d0 2024-03-29T11:39:42.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bd7fa640 10 client.4453.objecter ms_dispatch 0x7f64c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:42.739 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bffff640 10 monclient: get_auth_request con 0x7f64a406db60 auth_method 0 2024-03-29T11:39:42.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bd7fa640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:42.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.737+0000 7f64bd7fa640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:42.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.738+0000 7f64bd7fa640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:39:42.740 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.738+0000 7f64bd7fa640 10 client.4453.objecter ms_handle_connect 0x7f64a406db60 2024-03-29T11:39:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.743+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64b404ef30 con 0x7f64c00b17d0 2024-03-29T11:39:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.743+0000 7f64bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.745 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:42.743+0000 7f64bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.105+0000 7f64c7179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.105+0000 7f64c7179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.105+0000 7f64c7179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:39:43.107 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.105+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64a0005330 con 0x7f64c00b17d0 2024-03-29T11:39:43.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.107+0000 7f64bd7fa640 1 -- 172.21.15.45:0/1670035873 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f64b4051030 con 0x7f64c00b17d0 2024-03-29T11:39:43.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.107+0000 7f64bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:43.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.107+0000 7f64bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:43.109 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:43.109 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:43.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.110+0000 7f64c7179640 10 librados: watch_flush enter 2024-03-29T11:39:43.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 10 librados: watch_flush exit 2024-03-29T11:39:43.112 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 >> v1:172.21.15.190:6816/1411598462 conn(0x7f64a406db60 legacy=0x7f64a4070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 10 monclient: shutdown 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 >> v1:172.21.15.190:6789/0 conn(0x7f64c00b17d0 legacy=0x7f64c0132110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:43.113 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 shutdown_connections 2024-03-29T11:39:43.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 >> 172.21.15.45:0/1670035873 conn(0x7f64c0066a00 msgr2=0x7f64c013c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:43.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 shutdown_connections 2024-03-29T11:39:43.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.111+0000 7f64c7179640 1 -- 172.21.15.45:0/1670035873 wait complete. 2024-03-29T11:39:43.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:43.112+0000 7f64c7179640 1 librados: shutdown 2024-03-29T11:39:43.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:49.934 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.106+0000 7f17b5af6640 10 monclient: get_monmap_and_config 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.106+0000 7f17b5af6640 10 monclient: build_initial_monmap 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.106+0000 7f17b5af6640 10 monclient: monmap: 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:50.107661+0000 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:50.107661+0000 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:50.109 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.110 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:50.110 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:50.110 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:50.110 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.110 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.107+0000 7f17b5af6640 1 Processor -- start 2024-03-29T11:39:50.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.107+0000 7f17b5af6640 1 -- start start 2024-03-29T11:39:50.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.107+0000 7f17b5af6640 10 monclient: init 2024-03-29T11:39:50.111 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-a con 0x7f17b00b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-c con 0x7f17b014a700 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-b con 0x7f17b00ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00aba70 con 0x7f17b00b0770 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00aecb0 con 0x7f17b014a700 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00af150 con 0x7f17b00ac5a0 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.108+0000 7f17b5af6640 10 monclient(hunting): authenticate will time out at 1246.347707s 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17af7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f17b00b0770 0x7f17b0149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44239/0 (socket says 172.21.15.45:44239) 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17af7fe640 1 -- 172.21.15.45:0/3990005345 learned_addr learned my addr 172.21.15.45:0/3990005345 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:50.114 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214600513 0 0) 0x7f17b00aba70 con 0x7f17b00b0770 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.112+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003520 con 0x7f17b00b0770 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703996157 0 0) 0x7f17b00af150 con 0x7f17b00ac5a0 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.115 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:39:50.116 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b00aba70 con 0x7f17b00ac5a0 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432946039 0 0) 0x7f1798003520 con 0x7f17b00b0770 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 >> v1:172.21.15.45:6790/0 conn(0x7f17b00ac5a0 legacy=0x7f17b0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.113+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 >> v1:172.21.15.190:6789/0 conn(0x7f17b014a700 legacy=0x7f17b00abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 1 monclient: found mon.noname-a 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:50.117 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b00ac9d0 con 0x7f17b00b0770 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f17a4001690 con 0x7f17b00b0770 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.114+0000 7f17ae7fc640 10 monclient: dump: 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:50.118 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:50.119 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:50.119 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.117+0000 7f17b5af6640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:39:50.119 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.117+0000 7f17b5af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a40017f0 con 0x7f17b00b0770 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 1 -- 172.21.15.45:0/3990005345 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f17a4001c00 con 0x7f17b00b0770 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:50.121 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.118+0000 7f17ae7fc640 10 monclient: dump: 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:50.122 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.120+0000 7f17b5af6640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.120+0000 7f17b5af6640 4 set_mon_vals no callback set 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 10 monclient: shutdown 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 1 -- 172.21.15.45:0/3990005345 >> v1:172.21.15.45:6789/0 conn(0x7f17b00b0770 legacy=0x7f17b0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 1 -- 172.21.15.45:0/3990005345 shutdown_connections 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 1 -- 172.21.15.45:0/3990005345 >> 172.21.15.45:0/3990005345 conn(0x7f17b006b660 msgr2=0x7f17b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:50.123 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 1 -- 172.21.15.45:0/3990005345 shutdown_connections 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.121+0000 7f17b5af6640 1 -- 172.21.15.45:0/3990005345 wait complete. 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.122+0000 7f17b5af6640 10 monclient: build_initial_monmap 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.122+0000 7f17b5af6640 10 monclient: monmap: 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.125 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 librados: starting msgr at 2024-03-29T11:39:50.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 librados: starting objecter 2024-03-29T11:39:50.126 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 Processor -- start 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 -- start start 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 librados: setting wanted keys 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 1 librados: calling monclient init 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.123+0000 7f17b5af6640 10 monclient: init 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:50.127 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-a con 0x7f17b00ac5a0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-b con 0x7f17b00b0770 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient(hunting): picked mon.noname-c con 0x7f17b014a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b01285e0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b01dfa20 con 0x7f17b00b0770 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b0145060 con 0x7f17b014a700 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:50.128 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.124+0000 7f17b5af6640 10 monclient(hunting): authenticate will time out at 1246.363987s 2024-03-29T11:39:50.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.128+0000 7f17af7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f17b00ac5a0 0x7f17b01df310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:44499/0 (socket says 172.21.15.45:44499) 2024-03-29T11:39:50.130 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.128+0000 7f17af7fe640 1 -- 172.21.15.45:0/2579913511 learned_addr learned my addr 172.21.15.45:0/2579913511 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.128+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b00ac5a0 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.128+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401336512 0 0) 0x7f17b01285e0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:39:50.131 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.129+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c004eb0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840672568 0 0) 0x7f179c004eb0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f179c005300 con 0x7f17b00ac5a0 2024-03-29T11:39:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f17a4001b80 con 0x7f17b00ac5a0 2024-03-29T11:39:50.132 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient(hunting): dump: 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3150245119 0 0) 0x7f179c005300 con 0x7f17b00ac5a0 2024-03-29T11:39:50.133 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 >> v1:172.21.15.45:6790/0 conn(0x7f17b014a700 legacy=0x7f17b01262f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 >> v1:172.21.15.190:6789/0 conn(0x7f17b00b0770 legacy=0x7f17b01443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 monclient: found mon.a 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b01451f0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b00b0770 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b014a700 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.130+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 10 monclient: _renew_subs 2024-03-29T11:39:50.134 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.135 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b0144af0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:50.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 10 monclient: _renew_subs 2024-03-29T11:39:50.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b0126a00 con 0x7f17b00ac5a0 2024-03-29T11:39:50.138 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17b5af6640 1 librados: init done 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a4004a90 con 0x7f17b00ac5a0 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f1797fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.131+0000 7f17adffb640 4 set_mon_vals no callback set 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f17a4001b80 con 0x7f17b00ac5a0 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f1797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f1797fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f1797fff640 10 monclient: dump: 2024-03-29T11:39:50.139 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f17b5af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f17b5af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f17b5af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.132+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1790005080 con 0x7f17b00ac5a0 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.135+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f17a4004cc0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f17a4084670 con 0x7f17b00ac5a0 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f1797fff640 10 client.4488.objecter ms_dispatch 0x7f17b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:50.140 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f1797fff640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:50.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f1797fff640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:50.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f1797fff640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:39:50.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.136+0000 7f17aeffd640 10 monclient: get_auth_request con 0x7f179c06e2c0 auth_method 0 2024-03-29T11:39:50.141 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.137+0000 7f1797fff640 10 client.4488.objecter ms_handle_connect 0x7f179c06e2c0 2024-03-29T11:39:50.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.147+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17a404ffb0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.147+0000 7f1797fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:50.149 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.147+0000 7f1797fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:50.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.511+0000 7f17b5af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.511+0000 7f17b5af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.511+0000 7f17b5af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.511+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1790005330 con 0x7f17b00ac5a0 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.512+0000 7f1797fff640 1 -- 172.21.15.45:0/2579913511 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f17a404ffb0 con 0x7f17b00ac5a0 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.512+0000 7f1797fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.512+0000 7f1797fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:50.514 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:50.514 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:50.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.515+0000 7f17b5af6640 10 librados: watch_flush enter 2024-03-29T11:39:50.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.515+0000 7f17b5af6640 10 librados: watch_flush exit 2024-03-29T11:39:50.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.515+0000 7f17b5af6640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:39:50.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.515+0000 7f17b5af6640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:39:50.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.517+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 >> v1:172.21.15.190:6816/1411598462 conn(0x7f179c06e2c0 legacy=0x7f179c070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.517+0000 7f17b5af6640 10 monclient: shutdown 2024-03-29T11:39:50.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.517+0000 7f17b5af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:50.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.517+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 >> v1:172.21.15.45:6789/0 conn(0x7f17b00ac5a0 legacy=0x7f17b01df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.518+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 shutdown_connections 2024-03-29T11:39:50.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.518+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 >> 172.21.15.45:0/2579913511 conn(0x7f17b006b660 msgr2=0x7f17b00abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:50.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.518+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 shutdown_connections 2024-03-29T11:39:50.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.518+0000 7f17b5af6640 1 -- 172.21.15.45:0/2579913511 wait complete. 2024-03-29T11:39:50.520 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:50.519+0000 7f17b5af6640 1 librados: shutdown 2024-03-29T11:39:50.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:57.343 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:57.493 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.490+0000 7f596cfe8640 10 monclient: get_monmap_and_config 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.490+0000 7f596cfe8640 10 monclient: build_initial_monmap 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.490+0000 7f596cfe8640 10 monclient: monmap: 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:39:57.492029+0000 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:39:57.492029+0000 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:39:57.494 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:39:57.495 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.495 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.491+0000 7f596cfe8640 1 Processor -- start 2024-03-29T11:39:57.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.492+0000 7f596cfe8640 1 -- start start 2024-03-29T11:39:57.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.492+0000 7f596cfe8640 10 monclient: init 2024-03-29T11:39:57.497 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f59681422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-b con 0x7f5968142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-c con 0x7f59680b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968142c80 con 0x7f59681422b0 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59680aae40 con 0x7f59680b17d0 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596813fbd0 con 0x7f5968142690 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596cfe8640 10 monclient(hunting): authenticate will time out at 1253.732518s 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596659c640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f59681422b0 0x7f596813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60138/0 (socket says 172.21.15.45:60138) 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.493+0000 7f596659c640 1 -- 172.21.15.45:0/4046223269 learned_addr learned my addr 172.21.15.45:0/4046223269 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:57.498 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58294282 0 0) 0x7f5968142c80 con 0x7f59681422b0 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5954003520 con 0x7f59681422b0 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617061172 0 0) 0x7f59680aae40 con 0x7f59680b17d0 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968142c80 con 0x7f59680b17d0 2024-03-29T11:39:57.499 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777632065 0 0) 0x7f5954003520 con 0x7f59681422b0 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 >> v1:172.21.15.45:6790/0 conn(0x7f5968142690 legacy=0x7f59680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 >> v1:172.21.15.190:6789/0 conn(0x7f59680b17d0 legacy=0x7f59680b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 monclient: found mon.noname-a 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968151940 con 0x7f59681422b0 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: _finish_auth 0 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596cfe8640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596cfe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:57.500 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5958001690 con 0x7f59681422b0 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: dump: 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596cfe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:57.501 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59580018b0 con 0x7f59681422b0 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596cfe8640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.494+0000 7f596cfe8640 4 set_mon_vals no callback set 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596559a640 1 -- 172.21.15.45:0/4046223269 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5958001bc0 con 0x7f59681422b0 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596559a640 10 monclient: dump: 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.502 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 10 monclient: shutdown 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 1 -- 172.21.15.45:0/4046223269 >> v1:172.21.15.45:6789/0 conn(0x7f59681422b0 legacy=0x7f596813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 1 -- 172.21.15.45:0/4046223269 shutdown_connections 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 1 -- 172.21.15.45:0/4046223269 >> 172.21.15.45:0/4046223269 conn(0x7f5968066a00 msgr2=0x7f5968066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 1 -- 172.21.15.45:0/4046223269 shutdown_connections 2024-03-29T11:39:57.503 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.495+0000 7f596cfe8640 1 -- 172.21.15.45:0/4046223269 wait complete. 2024-03-29T11:39:57.507 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.496+0000 7f596cfe8640 10 monclient: build_initial_monmap 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.496+0000 7f596cfe8640 10 monclient: monmap: 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 librados: starting msgr at 2024-03-29T11:39:57.508 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 librados: starting objecter 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 Processor -- start 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 -- start start 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 librados: setting wanted keys 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 1 librados: calling monclient init 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.497+0000 7f596cfe8640 10 monclient: init 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-c con 0x7f59680b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f59680b17d0 0x7f59680ad0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:54716/0 (socket says 172.21.15.45:54716) 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f59681422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 10 monclient(hunting): picked mon.noname-b con 0x7f5968142690 addr v1:172.21.15.190:6789/0 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596659c640 1 -- 172.21.15.45:0/1768161400 learned_addr learned my addr 172.21.15.45:0/1768161400 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59680ac6c0 con 0x7f59681422b0 2024-03-29T11:39:57.509 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.498+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59680ac9a0 con 0x7f5968142690 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968071460 con 0x7f59680b17d0 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f596cfe8640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f596cfe8640 10 monclient(hunting): authenticate will time out at 1253.738149s 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter ms_handle_connect 0x7f59680b17d0 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter ms_handle_connect 0x7f59681422b0 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter ms_handle_connect 0x7f5968142690 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288024550 0 0) 0x7f59680ac6c0 con 0x7f59681422b0 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.510 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948004eb0 con 0x7f59681422b0 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765211827 0 0) 0x7f5968071460 con 0x7f59680b17d0 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59680ac6c0 con 0x7f59680b17d0 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090560558 0 0) 0x7f59680ac9a0 con 0x7f5968142690 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:39:57.511 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.499+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968071460 con 0x7f5968142690 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127052641 0 0) 0x7f59680ac6c0 con 0x7f59680b17d0 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59680ac9a0 con 0x7f59680b17d0 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5958001550 con 0x7f59680b17d0 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): dump: 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.512 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550304797 0 0) 0x7f5948004eb0 con 0x7f59681422b0 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59680ac6c0 con 0x7f59681422b0 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5950001790 con 0x7f59681422b0 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): dump: 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.513 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272382713 0 0) 0x7f5968071460 con 0x7f5968142690 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5948004eb0 con 0x7f5968142690 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5960001cb0 con 0x7f5968142690 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.500+0000 7f595effd640 10 monclient(hunting): dump: 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.514 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1348213156 0 0) 0x7f59680ac9a0 con 0x7f59680b17d0 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 >> v1:172.21.15.190:6789/0 conn(0x7f5968142690 legacy=0x7f59681ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 >> v1:172.21.15.45:6789/0 conn(0x7f59681422b0 legacy=0x7f59681da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 monclient: found mon.c 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968071f90 con 0x7f59680b17d0 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:57.515 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 10 monclient: _renew_subs 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59680ad7b0 con 0x7f59680b17d0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 10 monclient: _renew_subs 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59680ab7b0 con 0x7f59680b17d0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f596cfe8640 1 librados: init done 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5958001230 con 0x7f59680b17d0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5958004c30 con 0x7f59680b17d0 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.501+0000 7f595effd640 10 monclient: dump: 2024-03-29T11:39:57.516 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.502+0000 7f5964d99640 4 set_mon_vals no callback set 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.502+0000 7f596cfe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.503+0000 7f596cfe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.503+0000 7f596cfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.503+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c005080 con 0x7f59680b17d0 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.504+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f595801ac00 con 0x7f59680b17d0 2024-03-29T11:39:57.517 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f5958084aa0 con 0x7f59680b17d0 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f595effd640 10 client.4553.objecter ms_dispatch 0x7f5968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f595effd640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f5965d9b640 10 monclient: get_auth_request con 0x7f594806e0f0 auth_method 0 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f595effd640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.505+0000 7f595effd640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.506+0000 7f595effd640 10 client.4553.objecter ms_handle_connect 0x7f594806e0f0 2024-03-29T11:39:57.518 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.514+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5958050380 con 0x7f59680b17d0 2024-03-29T11:39:57.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.514+0000 7f595effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.519 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.514+0000 7f595effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:57.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.906+0000 7f596cfe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.906+0000 7f596cfe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.906+0000 7f596cfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:39:57.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.906+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f594c005330 con 0x7f59680b17d0 2024-03-29T11:39:57.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.907+0000 7f595effd640 1 -- 172.21.15.45:0/1768161400 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f59580522b0 con 0x7f59680b17d0 2024-03-29T11:39:57.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.907+0000 7f595effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.907+0000 7f595effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:57.910 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:39:57.910 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 10 librados: watch_flush enter 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 10 librados: watch_flush exit 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 >> v1:172.21.15.190:6816/1411598462 conn(0x7f594806e0f0 legacy=0x7f59480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 10 monclient: shutdown 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 >> v1:172.21.15.45:6790/0 conn(0x7f59680b17d0 legacy=0x7f59680ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 shutdown_connections 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 >> 172.21.15.45:0/1768161400 conn(0x7f5968066a00 msgr2=0x7f596813da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.913 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 shutdown_connections 2024-03-29T11:39:57.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 -- 172.21.15.45:0/1768161400 wait complete. 2024-03-29T11:39:57.914 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:39:57.911+0000 7f596cfe8640 1 librados: shutdown 2024-03-29T11:39:57.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:04.734 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.866+0000 7f04bfb7c640 10 monclient: get_monmap_and_config 2024-03-29T11:40:04.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.866+0000 7f04bfb7c640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.866+0000 7f04bfb7c640 10 monclient: monmap: 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:04.867997+0000 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:04.867997+0000 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:04.869 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.867+0000 7f04bfb7c640 1 Processor -- start 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.868+0000 7f04bfb7c640 1 -- start start 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.868+0000 7f04bfb7c640 10 monclient: init 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.868+0000 7f04bfb7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.868+0000 7f04bfb7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-b con 0x7f04b806ee40 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-c con 0x7f04b80aeda0 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-a con 0x7f04b813c470 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:04.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bd918640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f04b806ee40 0x7f04b813bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45454/0 (socket says 172.21.15.45:45454) 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bd918640 1 -- 172.21.15.45:0/261973136 learned_addr learned my addr 172.21.15.45:0/261973136 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b806fdb0 con 0x7f04b813c470 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8071090 con 0x7f04b80aeda0 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8071370 con 0x7f04b806ee40 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bfb7c640 10 monclient(hunting): authenticate will time out at 1261.108556s 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376696384 0 0) 0x7f04b8071090 con 0x7f04b80aeda0 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:40:04.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4005730 con 0x7f04b80aeda0 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155090270 0 0) 0x7f04b8071370 con 0x7f04b806ee40 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.869+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8071090 con 0x7f04b806ee40 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.870+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701991381 0 0) 0x7f04a4005730 con 0x7f04b80aeda0 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.870+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 >> v1:172.21.15.45:6790/0 conn(0x7f04b806ee40 legacy=0x7f04b813bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.870+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 >> v1:172.21.15.45:6789/0 conn(0x7f04b813c470 legacy=0x7f04b814a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 1 monclient: found mon.noname-c 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b8070220 con 0x7f04b80aeda0 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f04b4001690 con 0x7f04b80aeda0 2024-03-29T11:40:04.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bc916640 10 monclient: dump: 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:04.877 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bfb7c640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.872+0000 7f04bfb7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04b40017f0 con 0x7f04b80aeda0 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 1 -- 172.21.15.45:0/261973136 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f04b4001c40 con 0x7f04b80aeda0 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bc916640 10 monclient: dump: 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.878 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bfb7c640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:04.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.873+0000 7f04bfb7c640 4 set_mon_vals no callback set 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.874+0000 7f04bfb7c640 10 monclient: shutdown 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.874+0000 7f04bfb7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.874+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 >> v1:172.21.15.190:6789/0 conn(0x7f04b80aeda0 legacy=0x7f04b8071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.875+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 shutdown_connections 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.875+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 >> 172.21.15.45:0/261973136 conn(0x7f04b8067470 msgr2=0x7f04b80b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.876+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 shutdown_connections 2024-03-29T11:40:04.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.876+0000 7f04bfb7c640 1 -- 172.21.15.45:0/261973136 wait complete. 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.876+0000 7f04bfb7c640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.876+0000 7f04bfb7c640 10 monclient: monmap: 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:04.883 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 librados: starting msgr at 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 librados: starting objecter 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 Processor -- start 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 -- start start 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 librados: setting wanted keys 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 librados: calling monclient init 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient: init 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-c con 0x7f04b806ee40 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:04.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-b con 0x7f04b80aeda0 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient(hunting): picked mon.noname-a con 0x7f04b813c470 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8142d70 con 0x7f04b813c470 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8141a70 con 0x7f04b80aeda0 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8141520 con 0x7f04b806ee40 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.877+0000 7f04bfb7c640 10 monclient(hunting): authenticate will time out at 1261.117058s 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04bd918640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f04b806ee40 0x7f04b81d2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45852/0 (socket says 172.21.15.45:45852) 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04bd918640 1 -- 172.21.15.45:0/2315240640 learned_addr learned my addr 172.21.15.45:0/2315240640 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 client.?.objecter ms_handle_connect 0x7f04b806ee40 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261542401 0 0) 0x7f04b8141520 con 0x7f04b806ee40 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.878+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c004eb0 con 0x7f04b806ee40 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 701706680 0 0) 0x7f049c004eb0 con 0x7f04b806ee40 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b8141520 con 0x7f04b806ee40 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f04a8001a50 con 0x7f04b806ee40 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:04.886 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 client.?.objecter ms_handle_connect 0x7f04b80aeda0 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034159956 0 0) 0x7f04b8141a70 con 0x7f04b80aeda0 2024-03-29T11:40:04.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.879+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c004eb0 con 0x7f04b80aeda0 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.880+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3051503594 0 0) 0x7f04b8141520 con 0x7f04b806ee40 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.880+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 >> v1:172.21.15.190:6789/0 conn(0x7f04b80aeda0 legacy=0x7f04b81d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.880+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 >> v1:172.21.15.45:6789/0 conn(0x7f04b813c470 legacy=0x7f04b80ad750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:04.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 1 monclient: found mon.c 2024-03-29T11:40:04.891 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b81416b0 con 0x7f04b806ee40 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 10 monclient: _renew_subs 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b8140980 con 0x7f04b806ee40 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 10 monclient: _renew_subs 2024-03-29T11:40:04.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b80ae040 con 0x7f04b806ee40 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04bfb7c640 1 librados: init done 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04a8001cb0 con 0x7f04b806ee40 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f04a8004c30 con 0x7f04b806ee40 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04ae7fc640 10 monclient: dump: 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:04.893 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.881+0000 7f04affff640 4 set_mon_vals no callback set 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.882+0000 7f04bfb7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.882+0000 7f04bfb7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.882+0000 7f04bfb7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.882+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a0005080 con 0x7f04b806ee40 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.883+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f04a801ac00 con 0x7f04b806ee40 2024-03-29T11:40:04.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f04a8084b70 con 0x7f04b806ee40 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 10 client.4559.objecter ms_dispatch 0x7f04b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04bd117640 10 monclient: get_auth_request con 0x7f049c06dcb0 auth_method 0 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.884+0000 7f04ae7fc640 10 client.4559.objecter ms_handle_connect 0x7f049c06dcb0 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.891+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04a80504b0 con 0x7f04b806ee40 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.891+0000 7f04ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:04.891+0000 7f04ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:05.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.205+0000 7f04bfb7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:05.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.205+0000 7f04bfb7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:05.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.205+0000 7f04bfb7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:05.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.205+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04a0005330 con 0x7f04b806ee40 2024-03-29T11:40:05.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.206+0000 7f04ae7fc640 1 -- 172.21.15.45:0/2315240640 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f04a80523e0 con 0x7f04b806ee40 2024-03-29T11:40:05.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.206+0000 7f04ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:05.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.206+0000 7f04ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:05.209 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:05.209 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.209+0000 7f04bfb7c640 10 librados: watch_flush enter 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 10 librados: watch_flush exit 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 >> v1:172.21.15.190:6816/1411598462 conn(0x7f049c06dcb0 legacy=0x7f049c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 10 monclient: shutdown 2024-03-29T11:40:05.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 >> v1:172.21.15.45:6790/0 conn(0x7f04b806ee40 legacy=0x7f04b81d2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 shutdown_connections 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 >> 172.21.15.45:0/2315240640 conn(0x7f04b8067470 msgr2=0x7f04b813bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 shutdown_connections 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 -- 172.21.15.45:0/2315240640 wait complete. 2024-03-29T11:40:05.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:05.210+0000 7f04bfb7c640 1 librados: shutdown 2024-03-29T11:40:05.226 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:12.034 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:12.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.181+0000 7f2acca06640 10 monclient: get_monmap_and_config 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.181+0000 7f2acca06640 10 monclient: build_initial_monmap 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.181+0000 7f2acca06640 10 monclient: monmap: 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:12.182972+0000 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:12.182972+0000 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:12.184 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.182+0000 7f2acca06640 1 Processor -- start 2024-03-29T11:40:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.182+0000 7f2acca06640 1 -- start start 2024-03-29T11:40:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.182+0000 7f2acca06640 10 monclient: init 2024-03-29T11:40:12.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.183+0000 7f2acca06640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:12.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.183+0000 7f2acca06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:12.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.183+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac80b0770 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:12.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.188+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac814a700 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:12.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.188+0000 7f2abffff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2ac814a700 0x7f2ac80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:49017/0 (socket says 172.21.15.45:49017) 2024-03-29T11:40:12.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.189+0000 7f2abffff640 1 -- 172.21.15.45:0/3164161285 learned_addr learned my addr 172.21.15.45:0/3164161285 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:12.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.189+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac80ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:12.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.190+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80aba70 con 0x7f2ac814a700 2024-03-29T11:40:12.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.190+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80aecb0 con 0x7f2ac80b0770 2024-03-29T11:40:12.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.190+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80af150 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.190+0000 7f2acca06640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:12.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.190+0000 7f2acca06640 10 monclient(hunting): authenticate will time out at 1268.429313s 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793761107 0 0) 0x7f2ac80aba70 con 0x7f2ac814a700 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:40:12.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab4003520 con 0x7f2ac814a700 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104650228 0 0) 0x7f2ac80af150 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac80aba70 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211517864 0 0) 0x7f2ac80aecb0 con 0x7f2ac80b0770 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:40:12.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.192+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac80af150 con 0x7f2ac80b0770 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248693008 0 0) 0x7f2ac80aba70 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 >> v1:172.21.15.190:6789/0 conn(0x7f2ac80b0770 legacy=0x7f2ac8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 >> v1:172.21.15.45:6789/0 conn(0x7f2ac814a700 legacy=0x7f2ac80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80ac9d0 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2ac00018a0 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2ac5d9b640 10 monclient: dump: 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2acca06640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:40:12.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.193+0000 7f2acca06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:12.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac0001a00 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:12.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 1 -- 172.21.15.45:0/3164161285 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2ac0001e10 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:12.198 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.196+0000 7f2ac5d9b640 10 monclient: dump: 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.197+0000 7f2acca06640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:12.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.197+0000 7f2acca06640 4 set_mon_vals no callback set 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.197+0000 7f2acca06640 10 monclient: shutdown 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.197+0000 7f2acca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.197+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 >> v1:172.21.15.45:6790/0 conn(0x7f2ac80ac5a0 legacy=0x7f2ac8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.198+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 shutdown_connections 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.198+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 >> 172.21.15.45:0/3164161285 conn(0x7f2ac806b660 msgr2=0x7f2ac806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.199+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 shutdown_connections 2024-03-29T11:40:12.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.199+0000 7f2acca06640 1 -- 172.21.15.45:0/3164161285 wait complete. 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.200+0000 7f2acca06640 10 monclient: build_initial_monmap 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.200+0000 7f2acca06640 10 monclient: monmap: 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 librados: starting msgr at 2024-03-29T11:40:12.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 librados: starting objecter 2024-03-29T11:40:12.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 Processor -- start 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 -- start start 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 librados: setting wanted keys 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 1 librados: calling monclient init 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.201+0000 7f2acca06640 10 monclient: init 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2acca06640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2acca06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac80ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2ac659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2ac80ac5a0 0x7f2ac8145190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47028/0 (socket says 172.21.15.45:47028) 2024-03-29T11:40:12.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac80b0770 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.202+0000 7f2acca06640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac814a700 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2acca06640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8127f70 con 0x7f2ac814a700 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2acca06640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac81dea70 con 0x7f2ac80b0770 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2acca06640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac81281c0 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2acca06640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2acca06640 10 monclient(hunting): authenticate will time out at 1268.442186s 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2ac659c640 1 -- 172.21.15.45:0/4214778952 learned_addr learned my addr 172.21.15.45:0/4214778952 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2abeffd640 10 client.?.objecter ms_handle_connect 0x7f2ac80ac5a0 2024-03-29T11:40:12.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.203+0000 7f2abeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:12.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 client.?.objecter ms_handle_connect 0x7f2ac80b0770 2024-03-29T11:40:12.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:12.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936993614 0 0) 0x7f2ac81dea70 con 0x7f2ac80b0770 2024-03-29T11:40:12.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8004eb0 con 0x7f2ac80b0770 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927257205 0 0) 0x7f2ac81281c0 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:40:12.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.206+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac81dea70 con 0x7f2ac80ac5a0 2024-03-29T11:40:12.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 client.?.objecter ms_handle_connect 0x7f2ac814a700 2024-03-29T11:40:12.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:12.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895466358 0 0) 0x7f2aa8004eb0 con 0x7f2ac80b0770 2024-03-29T11:40:12.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac81281c0 con 0x7f2ac80b0770 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2ab8001890 con 0x7f2ac80b0770 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): dump: 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:12.214 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861960611 0 0) 0x7f2ac8127f70 con 0x7f2ac814a700 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8004eb0 con 0x7f2ac814a700 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2821226853 0 0) 0x7f2ac81281c0 con 0x7f2ac80b0770 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 >> v1:172.21.15.45:6790/0 conn(0x7f2ac80ac5a0 legacy=0x7f2ac8145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 >> v1:172.21.15.45:6789/0 conn(0x7f2ac814a700 legacy=0x7f2ac81e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 monclient: found mon.b 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac81284a0 con 0x7f2ac80b0770 2024-03-29T11:40:12.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.207+0000 7f2abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.208+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab80047e0 con 0x7f2ac80b0770 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.208+0000 7f2abeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.208+0000 7f2ac559a640 4 set_mon_vals no callback set 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.209+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2ab8004b30 con 0x7f2ac80b0770 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.209+0000 7f2abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.209+0000 7f2abeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.209+0000 7f2abeffd640 10 monclient: dump: 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:12.216 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 10 monclient: _renew_subs 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81239d0 con 0x7f2ac80b0770 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 10 monclient: _renew_subs 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac8145a80 con 0x7f2ac80b0770 2024-03-29T11:40:12.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.210+0000 7f2acca06640 1 librados: init done 2024-03-29T11:40:12.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.212+0000 7f2acca06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.212+0000 7f2acca06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.212+0000 7f2acca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.212+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80ab560 con 0x7f2ac80b0770 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.213+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f2ab8004c90 con 0x7f2ac80b0770 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.214+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2ab80848f0 con 0x7f2ac80b0770 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.214+0000 7f2abeffd640 10 client.4474.objecter ms_dispatch 0x7f2ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.214+0000 7f2abeffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.214+0000 7f2abeffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.214+0000 7f2abeffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.216+0000 7f2ac659c640 10 monclient: get_auth_request con 0x7f2aa806de80 auth_method 0 2024-03-29T11:40:12.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.216+0000 7f2abeffd640 10 client.4474.objecter ms_handle_connect 0x7f2aa806de80 2024-03-29T11:40:12.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.222+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ab80501d0 con 0x7f2ac80b0770 2024-03-29T11:40:12.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.222+0000 7f2abeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:12.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.222+0000 7f2abeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.550+0000 7f2acca06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.550+0000 7f2acca06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.550+0000 7f2acca06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.550+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ac8075f10 con 0x7f2ac80b0770 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.551+0000 7f2abeffd640 1 -- 172.21.15.45:0/4214778952 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f2ab8052100 con 0x7f2ac80b0770 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.551+0000 7f2abeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.551+0000 7f2abeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:12.554 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:12.554 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:12.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.554+0000 7f2acca06640 10 librados: watch_flush enter 2024-03-29T11:40:12.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.554+0000 7f2acca06640 10 librados: watch_flush exit 2024-03-29T11:40:12.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.554+0000 7f2acca06640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:40:12.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.554+0000 7f2acca06640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:40:12.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 >> v1:172.21.15.190:6816/1411598462 conn(0x7f2aa806de80 legacy=0x7f2aa8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 10 monclient: shutdown 2024-03-29T11:40:12.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 >> v1:172.21.15.190:6789/0 conn(0x7f2ac80b0770 legacy=0x7f2ac81def00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 shutdown_connections 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.555+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 >> 172.21.15.45:0/4214778952 conn(0x7f2ac806b660 msgr2=0x7f2ac806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.556+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 shutdown_connections 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.556+0000 7f2acca06640 1 -- 172.21.15.45:0/4214778952 wait complete. 2024-03-29T11:40:12.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:12.556+0000 7f2acca06640 1 librados: shutdown 2024-03-29T11:40:12.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:19.382 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.522+0000 7f4fef80f640 10 monclient: get_monmap_and_config 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.522+0000 7f4fef80f640 10 monclient: build_initial_monmap 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.522+0000 7f4fef80f640 10 monclient: monmap: 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:19.523956+0000 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:19.523956+0000 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:19.525 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.523+0000 7f4fef80f640 1 Processor -- start 2024-03-29T11:40:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.523+0000 7f4fef80f640 1 -- start start 2024-03-29T11:40:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.524+0000 7f4fef80f640 10 monclient: init 2024-03-29T11:40:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.525+0000 7f4fef80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:19.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.525+0000 7f4fef80f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:19.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.528+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe80b4790 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:19.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.529+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe80aea90 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:19.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.529+0000 7f4fed5ab640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4fe80b4790 0x7f4fe80b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:53611/0 (socket says 172.21.15.45:53611) 2024-03-29T11:40:19.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.529+0000 7f4fed5ab640 1 -- 172.21.15.45:0/4088059702 learned_addr learned my addr 172.21.15.45:0/4088059702 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.529+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80b39f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.530+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80b0c60 con 0x7f4fe80b39f0 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.530+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80af020 con 0x7f4fe80b4790 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.530+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe813fbd0 con 0x7f4fe80aea90 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.530+0000 7f4fef80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:19.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.530+0000 7f4fef80f640 10 monclient(hunting): authenticate will time out at 1275.769455s 2024-03-29T11:40:19.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202073322 0 0) 0x7f4fe813fbd0 con 0x7f4fe80aea90 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd4003520 con 0x7f4fe80aea90 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810391430 0 0) 0x7f4fe80af020 con 0x7f4fe80b4790 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:40:19.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe813fbd0 con 0x7f4fe80b4790 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228588383 0 0) 0x7f4fe813fbd0 con 0x7f4fe80b4790 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 >> v1:172.21.15.45:6790/0 conn(0x7f4fe80aea90 legacy=0x7f4fe80af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 >> v1:172.21.15.45:6789/0 conn(0x7f4fe80b39f0 legacy=0x7f4fe813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 monclient: found mon.noname-c 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe813fd60 con 0x7f4fe80b4790 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: _finish_auth 0 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:19.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4fe40015f0 con 0x7f4fe80b4790 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: dump: 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:19.539 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fef80f640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fef80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe4001750 con 0x7f4fe80b4790 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 1 -- 172.21.15.45:0/4088059702 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4fe4001ba0 con 0x7f4fe80b4790 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fdffff640 10 monclient: dump: 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.540 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fef80f640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.534+0000 7f4fef80f640 4 set_mon_vals no callback set 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.536+0000 7f4fef80f640 10 monclient: shutdown 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.536+0000 7f4fef80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.536+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 >> v1:172.21.15.190:6789/0 conn(0x7f4fe80b4790 legacy=0x7f4fe80b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.536+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 shutdown_connections 2024-03-29T11:40:19.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.536+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 >> 172.21.15.45:0/4088059702 conn(0x7f4fe8066a00 msgr2=0x7f4fe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.539+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 shutdown_connections 2024-03-29T11:40:19.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.540+0000 7f4fef80f640 1 -- 172.21.15.45:0/4088059702 wait complete. 2024-03-29T11:40:19.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 10 monclient: build_initial_monmap 2024-03-29T11:40:19.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 10 monclient: monmap: 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 librados: starting msgr at 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 librados: starting objecter 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 Processor -- start 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 -- start start 2024-03-29T11:40:19.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 librados: setting wanted keys 2024-03-29T11:40:19.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 1 librados: calling monclient init 2024-03-29T11:40:19.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.541+0000 7f4fef80f640 10 monclient: init 2024-03-29T11:40:19.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.542+0000 7f4fef80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:19.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.542+0000 7f4fef80f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.544+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe80b39f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe80b4790 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80704e0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80a97c0 con 0x7f4fe80704e0 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fed5ab640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4fe80b39f0 0x7f4fe81dbb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:53743/0 (socket says 172.21.15.45:53743) 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fed5ab640 1 -- 172.21.15.45:0/2036161721 learned_addr learned my addr 172.21.15.45:0/2036161721 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe81db620 con 0x7f4fe80b39f0 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe81dc250 con 0x7f4fe80b4790 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fef80f640 10 monclient(hunting): authenticate will time out at 1275.784516s 2024-03-29T11:40:19.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fddffb640 10 client.?.objecter ms_handle_connect 0x7f4fe80b39f0 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.545+0000 7f4fddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175352530 0 0) 0x7f4fe81db620 con 0x7f4fe80b39f0 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:40:19.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.546+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc004eb0 con 0x7f4fe80b39f0 2024-03-29T11:40:19.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 client.?.objecter ms_handle_connect 0x7f4fe80b4790 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297159148 0 0) 0x7f4fe81dc250 con 0x7f4fe80b4790 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe81db620 con 0x7f4fe80b4790 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134619909 0 0) 0x7f4fe81db620 con 0x7f4fe80b4790 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fcc0064f0 con 0x7f4fe80b4790 2024-03-29T11:40:19.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4fd8001b10 con 0x7f4fe80b4790 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): dump: 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.562 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414162384 0 0) 0x7f4fcc004eb0 con 0x7f4fe80b39f0 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fcc006bc0 con 0x7f4fe80b39f0 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4fe4001ba0 con 0x7f4fe80b39f0 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.551+0000 7f4fddffb640 10 monclient(hunting): dump: 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:19.563 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 603941995 0 0) 0x7f4fcc0064f0 con 0x7f4fe80b4790 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 >> v1:172.21.15.190:6789/0 conn(0x7f4fe80b39f0 legacy=0x7f4fe81dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 >> v1:172.21.15.45:6789/0 conn(0x7f4fe80704e0 legacy=0x7f4fe81e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 monclient: found mon.c 2024-03-29T11:40:19.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe8070100 con 0x7f4fe80b4790 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 10 monclient: _renew_subs 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe806de90 con 0x7f4fe80b4790 2024-03-29T11:40:19.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 10 monclient: _renew_subs 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe8070e10 con 0x7f4fe80b4790 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fef80f640 1 librados: init done 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd8001250 con 0x7f4fe80b4790 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.552+0000 7f4fdf7fe640 4 set_mon_vals no callback set 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4fd8001b10 con 0x7f4fe80b4790 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fddffb640 10 monclient: dump: 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:19.566 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fef80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fef80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fef80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.553+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8005080 con 0x7f4fe80b4790 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.555+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f4fd8019710 con 0x7f4fe80b4790 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.556+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f4fd8001930 con 0x7f4fe80b4790 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.556+0000 7f4fddffb640 10 client.4571.objecter ms_dispatch 0x7f4fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.556+0000 7f4fddffb640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.556+0000 7f4fddffb640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.556+0000 7f4fddffb640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:40:19.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.557+0000 7f4fed5ab640 10 monclient: get_auth_request con 0x7f4fcc06e5d0 auth_method 0 2024-03-29T11:40:19.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.561+0000 7f4fddffb640 10 client.4571.objecter ms_handle_connect 0x7f4fcc06e5d0 2024-03-29T11:40:19.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.562+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fd804f380 con 0x7f4fe80b4790 2024-03-29T11:40:19.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.562+0000 7f4fddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.562+0000 7f4fddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:19.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.882+0000 7f4fef80f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.882+0000 7f4fef80f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.882+0000 7f4fef80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:19.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.882+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc8005330 con 0x7f4fe80b4790 2024-03-29T11:40:19.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.883+0000 7f4fddffb640 1 -- 172.21.15.45:0/2036161721 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f4fd804f380 con 0x7f4fe80b4790 2024-03-29T11:40:19.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.883+0000 7f4fddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.883+0000 7f4fddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:19.886 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:19.886 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:19.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.887+0000 7f4fef80f640 10 librados: watch_flush enter 2024-03-29T11:40:19.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.889+0000 7f4fef80f640 10 librados: watch_flush exit 2024-03-29T11:40:19.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.889+0000 7f4fef80f640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:40:19.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.889+0000 7f4fef80f640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:40:19.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 >> v1:172.21.15.190:6816/1411598462 conn(0x7f4fcc06e5d0 legacy=0x7f4fcc070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 10 monclient: shutdown 2024-03-29T11:40:19.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 >> v1:172.21.15.45:6790/0 conn(0x7f4fe80b4790 legacy=0x7f4fe8134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 shutdown_connections 2024-03-29T11:40:19.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.890+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 >> 172.21.15.45:0/2036161721 conn(0x7f4fe8066a00 msgr2=0x7f4fe80b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.894+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 shutdown_connections 2024-03-29T11:40:19.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.894+0000 7f4fef80f640 1 -- 172.21.15.45:0/2036161721 wait complete. 2024-03-29T11:40:19.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:19.894+0000 7f4fef80f640 1 librados: shutdown 2024-03-29T11:40:19.912 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:26.721 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:26.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.865+0000 7f6f0d6d6640 10 monclient: get_monmap_and_config 2024-03-29T11:40:26.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.865+0000 7f6f0d6d6640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.865+0000 7f6f0d6d6640 10 monclient: monmap: 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:26.865759+0000 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:26.865759+0000 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:26.867 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.867+0000 7f6f0d6d6640 1 Processor -- start 2024-03-29T11:40:26.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.867+0000 7f6f0d6d6640 1 -- start start 2024-03-29T11:40:26.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.867+0000 7f6f0d6d6640 10 monclient: init 2024-03-29T11:40:26.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.868+0000 7f6f0d6d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.868+0000 7f6f0d6d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:26.870 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6f080b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0814a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6f080ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080aba70 con 0x7f6f080b0770 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080aecb0 con 0x7f6f080ac5a0 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080af150 con 0x7f6f0814a700 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f067fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6f0814a700 0x7f6f080abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:60486/0 (socket says 172.21.15.45:60486) 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f067fc640 1 -- 172.21.15.45:0/4287494661 learned_addr learned my addr 172.21.15.45:0/4287494661 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:26.871 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.869+0000 7f6f0d6d6640 10 monclient(hunting): authenticate will time out at 1283.108048s 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118913532 0 0) 0x7f6f080af150 con 0x7f6f0814a700 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:40:26.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.872+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef0003520 con 0x7f6f0814a700 2024-03-29T11:40:26.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1854386683 0 0) 0x7f6ef0003520 con 0x7f6f0814a700 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 >> v1:172.21.15.190:6789/0 conn(0x7f6f080ac5a0 legacy=0x7f6f08151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 >> v1:172.21.15.45:6789/0 conn(0x7f6f080b0770 legacy=0x7f6f08149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 monclient: found mon.noname-b 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080ac9d0 con 0x7f6f0814a700 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc001690 con 0x7f6f0814a700 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.875+0000 7f6f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:26.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: dump: 2024-03-29T11:40:26.878 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:26.878 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:26.878 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:26.879 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f0d6d6640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f0d6d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc0018b0 con 0x7f6f0814a700 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 1 -- 172.21.15.45:0/4287494661 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc001c40 con 0x7f6f0814a700 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f05ffb640 10 monclient: dump: 2024-03-29T11:40:26.880 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:26.881 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:26.882 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f0d6d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.876+0000 7f6f0d6d6640 4 set_mon_vals no callback set 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 10 monclient: shutdown 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/4287494661 >> v1:172.21.15.45:6790/0 conn(0x7f6f0814a700 legacy=0x7f6f080abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/4287494661 shutdown_connections 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/4287494661 >> 172.21.15.45:0/4287494661 conn(0x7f6f0806b660 msgr2=0x7f6f0806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/4287494661 shutdown_connections 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.877+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/4287494661 wait complete. 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.878+0000 7f6f0d6d6640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.878+0000 7f6f0d6d6640 10 monclient: monmap: 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:26.895 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.878+0000 7f6f0d6d6640 1 librados: starting msgr at 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.878+0000 7f6f0d6d6640 1 librados: starting objecter 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 1 Processor -- start 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 1 -- start start 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 1 librados: setting wanted keys 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 1 librados: calling monclient init 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 10 monclient: init 2024-03-29T11:40:26.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.879+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f080ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6f080b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0814a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f067fc640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6f080b0770 0x7f6f08126c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:52821/0 (socket says 172.21.15.45:52821) 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08126090 con 0x7f6f080b0770 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08125c40 con 0x7f6f080ac5a0 2024-03-29T11:40:26.898 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08125dd0 con 0x7f6f0814a700 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f0d6d6640 10 monclient(hunting): authenticate will time out at 1283.120598s 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6f067fc640 1 -- 172.21.15.45:0/1384492608 learned_addr learned my addr 172.21.15.45:0/1384492608 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 client.?.objecter ms_handle_connect 0x7f6f0814a700 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 client.?.objecter ms_handle_connect 0x7f6f080b0770 2024-03-29T11:40:26.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474879073 0 0) 0x7f6f08126090 con 0x7f6f080b0770 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.882+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4004eb0 con 0x7f6f080b0770 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625452443 0 0) 0x7f6f08125dd0 con 0x7f6f0814a700 2024-03-29T11:40:26.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f08126090 con 0x7f6f0814a700 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199671638 0 0) 0x7f6ef4004eb0 con 0x7f6f080b0770 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f08125dd0 con 0x7f6f080b0770 2024-03-29T11:40:26.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc001bc0 con 0x7f6f080b0770 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.902 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:26.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2853942285 0 0) 0x7f6f08125dd0 con 0x7f6f080b0770 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 >> v1:172.21.15.45:6790/0 conn(0x7f6f0814a700 legacy=0x7f6f081e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 >> v1:172.21.15.190:6789/0 conn(0x7f6f080ac5a0 legacy=0x7f6f08126520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 monclient: found mon.a 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08145a50 con 0x7f6f080b0770 2024-03-29T11:40:26.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6f0d6d6640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6f0d6d6640 10 monclient: _renew_subs 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6f0d6d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:26.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.883+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc004a90 con 0x7f6f080b0770 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f081dbfb0 con 0x7f6f080b0770 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6eef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc004da0 con 0x7f6f080b0770 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6eef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6eef7fe640 10 monclient: dump: 2024-03-29T11:40:26.906 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:26.907 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f057fa640 4 set_mon_vals no callback set 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 10 monclient: _renew_subs 2024-03-29T11:40:26.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f081db770 con 0x7f6f080b0770 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.884+0000 7f6f0d6d6640 1 librados: init done 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.886+0000 7f6f0d6d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.886+0000 7f6f0d6d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.886+0000 7f6f0d6d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.886+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f080ab560 con 0x7f6f080b0770 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.886+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f6efc01ad70 con 0x7f6f080b0770 2024-03-29T11:40:26.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f6efc084b90 con 0x7f6f080b0770 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6eef7fe640 10 client.4509.objecter ms_dispatch 0x7f6f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6eef7fe640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6eef7fe640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6eef7fe640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.887+0000 7f6f06ffd640 10 monclient: get_auth_request con 0x7f6ef406ddd0 auth_method 0 2024-03-29T11:40:26.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.888+0000 7f6eef7fe640 10 client.4509.objecter ms_handle_connect 0x7f6ef406ddd0 2024-03-29T11:40:26.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.895+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6efc050470 con 0x7f6f080b0770 2024-03-29T11:40:26.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.895+0000 7f6eef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.911 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:26.895+0000 7f6eef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:27.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.223+0000 7f6f0d6d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:27.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.223+0000 7f6f0d6d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:27.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.223+0000 7f6f0d6d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:27.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.223+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f08075f10 con 0x7f6f080b0770 2024-03-29T11:40:27.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.224+0000 7f6eef7fe640 1 -- 172.21.15.45:0/1384492608 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f6efc0523a0 con 0x7f6f080b0770 2024-03-29T11:40:27.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.224+0000 7f6eef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:27.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.224+0000 7f6eef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:27.227 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:27.227 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:27.229 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.228+0000 7f6f0d6d6640 10 librados: watch_flush enter 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 10 librados: watch_flush exit 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 >> v1:172.21.15.190:6816/1411598462 conn(0x7f6ef406ddd0 legacy=0x7f6ef4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 10 monclient: shutdown 2024-03-29T11:40:27.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 >> v1:172.21.15.45:6789/0 conn(0x7f6f080b0770 legacy=0x7f6f08126c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 shutdown_connections 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.229+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 >> 172.21.15.45:0/1384492608 conn(0x7f6f0806b660 msgr2=0x7f6f0806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.230+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 shutdown_connections 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.230+0000 7f6f0d6d6640 1 -- 172.21.15.45:0/1384492608 wait complete. 2024-03-29T11:40:27.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:27.230+0000 7f6f0d6d6640 1 librados: shutdown 2024-03-29T11:40:27.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:34.054 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.198+0000 7f2dae90e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.198+0000 7f2dae90e640 10 monclient: build_initial_monmap 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.198+0000 7f2dae90e640 10 monclient: monmap: 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:34.199046+0000 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:34.199046+0000 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:34.200 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.201 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:34.201 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:34.201 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:34.201 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.201+0000 7f2dae90e640 1 Processor -- start 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.201+0000 7f2dae90e640 1 -- start start 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.201+0000 7f2dae90e640 10 monclient: init 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:34.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-a con 0x7f2da80b1f60 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-b con 0x7f2da8071eb0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-c con 0x7f2da8143d10 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da806ef80 con 0x7f2da80b1f60 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da806f260 con 0x7f2da8143d10 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80b40c0 con 0x7f2da8071eb0 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.202+0000 7f2dae90e640 10 monclient(hunting): authenticate will time out at 1290.440770s 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.203+0000 7f2da77fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2da8071eb0 0x7f2da8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57764/0 (socket says 172.21.15.45:57764) 2024-03-29T11:40:34.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.203+0000 7f2da77fe640 1 -- 172.21.15.45:0/4129528177 learned_addr learned my addr 172.21.15.45:0/4129528177 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:34.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416796111 0 0) 0x7f2da80b40c0 con 0x7f2da8071eb0 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d90003520 con 0x7f2da8071eb0 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2527297091 0 0) 0x7f2d90003520 con 0x7f2da8071eb0 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.204+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 >> v1:172.21.15.190:6789/0 conn(0x7f2da8143d10 legacy=0x7f2da8146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:34.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.205+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 >> v1:172.21.15.45:6789/0 conn(0x7f2da80b1f60 legacy=0x7f2da8143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8155d70 con 0x7f2da8071eb0 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:34.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2d9c001690 con 0x7f2da8071eb0 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: dump: 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:34.209 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2dae90e640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2dae90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d9c001860 con 0x7f2da8071eb0 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 1 -- 172.21.15.45:0/4129528177 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2d9c001cb0 con 0x7f2da8071eb0 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2da6ffd640 10 monclient: dump: 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.210 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:34.211 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:34.211 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:34.211 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2dae90e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:34.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.206+0000 7f2dae90e640 4 set_mon_vals no callback set 2024-03-29T11:40:34.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.207+0000 7f2dae90e640 10 monclient: shutdown 2024-03-29T11:40:34.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.207+0000 7f2dae90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:34.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.207+0000 7f2dae90e640 1 -- 172.21.15.45:0/4129528177 >> v1:172.21.15.45:6790/0 conn(0x7f2da8071eb0 legacy=0x7f2da8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.208+0000 7f2dae90e640 1 -- 172.21.15.45:0/4129528177 shutdown_connections 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.208+0000 7f2dae90e640 1 -- 172.21.15.45:0/4129528177 >> 172.21.15.45:0/4129528177 conn(0x7f2da8066a00 msgr2=0x7f2da8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.208+0000 7f2dae90e640 1 -- 172.21.15.45:0/4129528177 shutdown_connections 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.208+0000 7f2dae90e640 1 -- 172.21.15.45:0/4129528177 wait complete. 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.209+0000 7f2dae90e640 10 monclient: build_initial_monmap 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.209+0000 7f2dae90e640 10 monclient: monmap: 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.214 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 librados: starting msgr at 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 librados: starting objecter 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 Processor -- start 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 -- start start 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 librados: setting wanted keys 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 1 librados: calling monclient init 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.210+0000 7f2dae90e640 10 monclient: init 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.211+0000 7f2dae90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.211+0000 7f2dae90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:34.215 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.211+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-c con 0x7f2da80b1f60 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:34.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.214+0000 7f2da7fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f2da80b1f60 0x7f2da80aa6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58566/0 (socket says 172.21.15.45:58566) 2024-03-29T11:40:34.216 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.215+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-b con 0x7f2da8143d10 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:34.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.216+0000 7f2da7fff640 1 -- 172.21.15.45:0/1538415966 learned_addr learned my addr 172.21.15.45:0/1538415966 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:34.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.216+0000 7f2da4ff9640 10 client.?.objecter ms_handle_connect 0x7f2da80b1f60 2024-03-29T11:40:34.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.216+0000 7f2da4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:34.217 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.216+0000 7f2dae90e640 10 monclient(hunting): picked mon.noname-a con 0x7f2da80aa070 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:34.218 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.216+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80aaee0 con 0x7f2da80aa070 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2da4ff9640 10 client.?.objecter ms_handle_connect 0x7f2da8143d10 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2da4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80ab370 con 0x7f2da8143d10 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80ac1f0 con 0x7f2da80b1f60 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2dae90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897675409 0 0) 0x7f2da80ac1f0 con 0x7f2da80b1f60 2024-03-29T11:40:34.219 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.217+0000 7f2dae90e640 10 monclient(hunting): authenticate will time out at 1290.455925s 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94004eb0 con 0x7f2da80b1f60 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404860672 0 0) 0x7f2da80ab370 con 0x7f2da8143d10 2024-03-29T11:40:34.224 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da80ac1f0 con 0x7f2da8143d10 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625671928 0 0) 0x7f2d94004eb0 con 0x7f2da80b1f60 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da80ab370 con 0x7f2da80b1f60 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.218+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2da0001ce0 con 0x7f2da80b1f60 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.225 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787301004 0 0) 0x7f2da80ab370 con 0x7f2da80b1f60 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 >> v1:172.21.15.190:6789/0 conn(0x7f2da8143d10 legacy=0x7f2da806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 >> v1:172.21.15.45:6789/0 conn(0x7f2da80aa070 legacy=0x7f2da81ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 monclient: found mon.c 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:34.226 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da80ac4d0 con 0x7f2da80b1f60 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2da0003970 con 0x7f2da80b1f60 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2da0003cc0 con 0x7f2da80b1f60 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.219+0000 7f2da4ff9640 10 monclient: dump: 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:34.227 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 10 monclient: _renew_subs 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da806fff0 con 0x7f2da80b1f60 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.220+0000 7f2dae90e640 10 monclient: _renew_subs 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.221+0000 7f2dae90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.221+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da81efd20 con 0x7f2da80b1f60 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.221+0000 7f2dae90e640 1 librados: init done 2024-03-29T11:40:34.228 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.221+0000 7f2da67fc640 4 set_mon_vals no callback set 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.223+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f2da0003e20 con 0x7f2da80b1f60 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.223+0000 7f2dae90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.223+0000 7f2dae90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.223+0000 7f2dae90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.223+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d88005080 con 0x7f2da80b1f60 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.224+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2da0084da0 con 0x7f2da80b1f60 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.224+0000 7f2da4ff9640 10 client.4583.objecter ms_dispatch 0x7f2da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.224+0000 7f2da4ff9640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:34.230 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.224+0000 7f2da4ff9640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:34.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.225+0000 7f2da4ff9640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:40:34.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.225+0000 7f2da77fe640 10 monclient: get_auth_request con 0x7f2d9406dd60 auth_method 0 2024-03-29T11:40:34.231 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.225+0000 7f2da4ff9640 10 client.4583.objecter ms_handle_connect 0x7f2d9406dd60 2024-03-29T11:40:34.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.231+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2da00506e0 con 0x7f2da80b1f60 2024-03-29T11:40:34.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.231+0000 7f2da4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:34.233 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.231+0000 7f2da4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:34.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.575+0000 7f2dae90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.577 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.575+0000 7f2dae90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.575+0000 7f2dae90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:34.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.575+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d88005330 con 0x7f2da80b1f60 2024-03-29T11:40:34.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.576+0000 7f2da4ff9640 1 -- 172.21.15.45:0/1538415966 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f2da0084450 con 0x7f2da80b1f60 2024-03-29T11:40:34.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.576+0000 7f2da4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.576+0000 7f2da4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:34.582 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:34.582 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:34.583 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.582+0000 7f2dae90e640 10 librados: watch_flush enter 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.582+0000 7f2dae90e640 10 librados: watch_flush exit 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.582+0000 7f2dae90e640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.582+0000 7f2dae90e640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 >> v1:172.21.15.190:6816/1411598462 conn(0x7f2d9406dd60 legacy=0x7f2d94070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 10 monclient: shutdown 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 >> v1:172.21.15.45:6790/0 conn(0x7f2da80b1f60 legacy=0x7f2da80aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 shutdown_connections 2024-03-29T11:40:34.584 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 >> 172.21.15.45:0/1538415966 conn(0x7f2da8066a00 msgr2=0x7f2da8146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:34.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.583+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 shutdown_connections 2024-03-29T11:40:34.585 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.584+0000 7f2dae90e640 1 -- 172.21.15.45:0/1538415966 wait complete. 2024-03-29T11:40:34.586 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:34.585+0000 7f2dae90e640 1 librados: shutdown 2024-03-29T11:40:34.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:41.410 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:41.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.540+0000 7ffa798e2640 10 monclient: get_monmap_and_config 2024-03-29T11:40:41.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.540+0000 7ffa798e2640 10 monclient: build_initial_monmap 2024-03-29T11:40:41.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.540+0000 7ffa798e2640 10 monclient: monmap: 2024-03-29T11:40:41.542 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:41.542 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:41.541393+0000 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:41.541393+0000 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:41.543 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.541+0000 7ffa798e2640 1 Processor -- start 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.541+0000 7ffa798e2640 1 -- start start 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.541+0000 7ffa798e2640 10 monclient: init 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.542+0000 7ffa798e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.542+0000 7ffa798e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:41.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.542+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-a con 0x7ffa741457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:41.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.542+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7413f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:41.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.542+0000 7ffa727fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ffa7413f650 0x7ffa740b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:44058/0 (socket says 172.21.15.45:44058) 2024-03-29T11:40:41.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-c con 0x7ffa740a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:41.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa727fc640 1 -- 172.21.15.45:0/549301532 learned_addr learned my addr 172.21.15.45:0/549301532 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa740aaef0 con 0x7ffa741457f0 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74145bd0 con 0x7ffa740a9d70 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74145eb0 con 0x7ffa7413f650 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa798e2640 10 monclient(hunting): authenticate will time out at 1297.782307s 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793888706 0 0) 0x7ffa740aaef0 con 0x7ffa741457f0 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c003520 con 0x7ffa741457f0 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219083517 0 0) 0x7ffa74145eb0 con 0x7ffa7413f650 2024-03-29T11:40:41.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa740aaef0 con 0x7ffa7413f650 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123798908 0 0) 0x7ffa5c003520 con 0x7ffa741457f0 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 >> v1:172.21.15.45:6790/0 conn(0x7ffa7413f650 legacy=0x7ffa740b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.543+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 >> v1:172.21.15.190:6789/0 conn(0x7ffa740a9d70 legacy=0x7ffa7414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 1 monclient: found mon.noname-a 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74146340 con 0x7ffa741457f0 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:41.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa6c001940 con 0x7ffa741457f0 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa71ffb640 10 monclient: dump: 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:41.552 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa798e2640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.544+0000 7ffa798e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa6c001b60 con 0x7ffa741457f0 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 1 -- 172.21.15.45:0/549301532 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa6c001ef0 con 0x7ffa741457f0 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa71ffb640 10 monclient: dump: 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.553 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 4 set_mon_vals no callback set 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 10 monclient: shutdown 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 >> v1:172.21.15.45:6789/0 conn(0x7ffa741457f0 legacy=0x7ffa7413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 shutdown_connections 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.545+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 >> 172.21.15.45:0/549301532 conn(0x7ffa74066a00 msgr2=0x7ffa74066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.546+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 shutdown_connections 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.546+0000 7ffa798e2640 1 -- 172.21.15.45:0/549301532 wait complete. 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.546+0000 7ffa798e2640 10 monclient: build_initial_monmap 2024-03-29T11:40:41.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.546+0000 7ffa798e2640 10 monclient: monmap: 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 librados: starting msgr at 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 librados: starting objecter 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 Processor -- start 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 -- start start 2024-03-29T11:40:41.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 librados: setting wanted keys 2024-03-29T11:40:41.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 1 librados: calling monclient init 2024-03-29T11:40:41.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.547+0000 7ffa798e2640 10 monclient: init 2024-03-29T11:40:41.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:41.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:41.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-a con 0x7ffa740a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7413f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient(hunting): picked mon.noname-c con 0x7ffa741457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa741dac90 con 0x7ffa740a9d70 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa741359f0 con 0x7ffa7413f650 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74135cd0 con 0x7ffa741457f0 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa798e2640 10 monclient(hunting): authenticate will time out at 1297.787565s 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa72ffd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7ffa740a9d70 0x7ffa741db120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:40258/0 (socket says 172.21.15.45:40258) 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa72ffd640 1 -- 172.21.15.45:0/1102322358 learned_addr learned my addr 172.21.15.45:0/1102322358 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.548+0000 7ffa5bfff640 10 client.?.objecter ms_handle_connect 0x7ffa740a9d70 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 client.?.objecter ms_handle_connect 0x7ffa741457f0 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:41.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320585565 0 0) 0x7ffa74135cd0 con 0x7ffa741457f0 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60004eb0 con 0x7ffa741457f0 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.549+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325879064 0 0) 0x7ffa60004eb0 con 0x7ffa741457f0 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa74135cd0 con 0x7ffa741457f0 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa680017f0 con 0x7ffa741457f0 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): dump: 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.562 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996479060 0 0) 0x7ffa741dac90 con 0x7ffa740a9d70 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60004eb0 con 0x7ffa740a9d70 2024-03-29T11:40:41.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988321972 0 0) 0x7ffa74135cd0 con 0x7ffa741457f0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.550+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 >> v1:172.21.15.190:6789/0 conn(0x7ffa7413f650 legacy=0x7ffa7413fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 >> v1:172.21.15.45:6789/0 conn(0x7ffa740a9d70 legacy=0x7ffa741db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 1 monclient: found mon.c 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74134ae0 con 0x7ffa741457f0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 10 monclient: _renew_subs 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:41.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa741da7b0 con 0x7ffa741457f0 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 10 monclient: _renew_subs 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa74139c90 con 0x7ffa741457f0 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.552+0000 7ffa798e2640 1 librados: init done 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa68001d80 con 0x7ffa741457f0 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ffa68003830 con 0x7ffa741457f0 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa5bfff640 10 monclient: dump: 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.565 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.553+0000 7ffa717fa640 4 set_mon_vals no callback set 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.554+0000 7ffa798e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.554+0000 7ffa798e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.554+0000 7ffa798e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.554+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa54005080 con 0x7ffa741457f0 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.555+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7ffa68019800 con 0x7ffa741457f0 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.555+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7ffa68083820 con 0x7ffa741457f0 2024-03-29T11:40:41.566 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.555+0000 7ffa5bfff640 10 client.4589.objecter ms_dispatch 0x7ffa74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.555+0000 7ffa5bfff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.555+0000 7ffa5bfff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.556+0000 7ffa5bfff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.556+0000 7ffa72ffd640 10 monclient: get_auth_request con 0x7ffa6006dad0 auth_method 0 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.556+0000 7ffa5bfff640 10 client.4589.objecter ms_handle_connect 0x7ffa6006dad0 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.563+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa6804ef30 con 0x7ffa741457f0 2024-03-29T11:40:41.567 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.563+0000 7ffa5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.568 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.563+0000 7ffa5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:41.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.879+0000 7ffa798e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.879+0000 7ffa798e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.879+0000 7ffa798e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:41.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.879+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa54005330 con 0x7ffa741457f0 2024-03-29T11:40:41.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.880+0000 7ffa5bfff640 1 -- 172.21.15.45:0/1102322358 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7ffa68051030 con 0x7ffa741457f0 2024-03-29T11:40:41.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.880+0000 7ffa5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.880+0000 7ffa5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:41.883 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:41.883 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:41.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 10 librados: watch_flush enter 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 10 librados: watch_flush exit 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 >> v1:172.21.15.190:6816/1411598462 conn(0x7ffa6006dad0 legacy=0x7ffa6006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 10 monclient: shutdown 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.885+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 >> v1:172.21.15.45:6790/0 conn(0x7ffa741457f0 legacy=0x7ffa741e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.886+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 shutdown_connections 2024-03-29T11:40:41.888 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.886+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 >> 172.21.15.45:0/1102322358 conn(0x7ffa74066a00 msgr2=0x7ffa7413ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.886+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 shutdown_connections 2024-03-29T11:40:41.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.886+0000 7ffa798e2640 1 -- 172.21.15.45:0/1102322358 wait complete. 2024-03-29T11:40:41.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:41.886+0000 7ffa798e2640 1 librados: shutdown 2024-03-29T11:40:41.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:48.713 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:48.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.853+0000 7f3d40832640 10 monclient: get_monmap_and_config 2024-03-29T11:40:48.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.853+0000 7f3d40832640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.855 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.853+0000 7f3d40832640 10 monclient: monmap: 2024-03-29T11:40:48.855 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:48.854247+0000 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:48.854247+0000 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:48.856 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.853+0000 7f3d40832640 1 Processor -- start 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 1 -- start start 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient: init 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-c con 0x7f3d3c0b4820 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-b con 0x7f3d3c0a9ce0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3c0b3cd0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0b4180 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0aaf60 con 0x7f3d3c0b4820 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c13c200 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.861 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.854+0000 7f3d40832640 10 monclient(hunting): authenticate will time out at 1305.094014s 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d3b59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f3d3c0b3cd0 0x7f3d3c14d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:45558/0 (socket says 172.21.15.45:45558) 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d3b59e640 1 -- 172.21.15.45:0/1574505404 learned_addr learned my addr 172.21.15.45:0/1574505404 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525293084 0 0) 0x7f3d2c001110 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d24003520 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489388668 0 0) 0x7f3d3c13c200 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:40:48.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.855+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d2c001110 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398227686 0 0) 0x7f3d24003520 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 >> v1:172.21.15.45:6790/0 conn(0x7f3d3c0a9ce0 legacy=0x7f3d3c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 >> v1:172.21.15.190:6789/0 conn(0x7f3d3c0b4820 legacy=0x7f3d3c13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 monclient: found mon.noname-a 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c145e80 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3d2c001680 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d40832640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d40832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: dump: 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.864 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d40832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d2c0017e0 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d39d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d40832640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.856+0000 7f3d40832640 4 set_mon_vals no callback set 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d39d9b640 1 -- 172.21.15.45:0/1574505404 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3d2c001bf0 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d39d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:48.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d40832640 10 monclient: shutdown 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d39d9b640 10 monclient: dump: 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d40832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d40832640 1 -- 172.21.15.45:0/1574505404 >> v1:172.21.15.45:6789/0 conn(0x7f3d3c0b3cd0 legacy=0x7f3d3c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d40832640 1 -- 172.21.15.45:0/1574505404 shutdown_connections 2024-03-29T11:40:48.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.857+0000 7f3d40832640 1 -- 172.21.15.45:0/1574505404 >> 172.21.15.45:0/1574505404 conn(0x7f3d3c06d070 msgr2=0x7f3d3c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.858+0000 7f3d40832640 1 -- 172.21.15.45:0/1574505404 shutdown_connections 2024-03-29T11:40:48.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.858+0000 7f3d40832640 1 -- 172.21.15.45:0/1574505404 wait complete. 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 10 monclient: monmap: 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:48.874 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 librados: starting msgr at 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 librados: starting objecter 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 Processor -- start 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 -- start start 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 librados: setting wanted keys 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 1 librados: calling monclient init 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.859+0000 7f3d40832640 10 monclient: init 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.860+0000 7f3d40832640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.860+0000 7f3d40832640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.860+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-c con 0x7f3d3c0a9ce0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.860+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3c0b3cd0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.860+0000 7f3d40832640 10 monclient(hunting): picked mon.noname-b con 0x7f3d3c0b4820 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d40832640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c06ef30 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d40832640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c06f210 con 0x7f3d3c0b4820 2024-03-29T11:40:48.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d40832640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c144fe0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d40832640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d40832640 10 monclient(hunting): authenticate will time out at 1305.100213s 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d3b59e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3d3c0b4820 0x7f3d3c1448d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:39330/0 (socket says 172.21.15.45:39330) 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d3b59e640 1 -- 172.21.15.45:0/1761849267 learned_addr learned my addr 172.21.15.45:0/1761849267 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter ms_handle_connect 0x7f3d3c0b4820 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241954254 0 0) 0x7f3d3c06f210 con 0x7f3d3c0b4820 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28004eb0 con 0x7f3d3c0b4820 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter ms_handle_connect 0x7f3d3c0b3cd0 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter ms_handle_connect 0x7f3d3c0a9ce0 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.861+0000 7f3d237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94703966 0 0) 0x7f3d28004eb0 con 0x7f3d3c0b4820 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d3c06f210 con 0x7f3d3c0b4820 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3d2c0047e0 con 0x7f3d3c0b4820 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): dump: 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:48.877 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331264287 0 0) 0x7f3d3c144fe0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28004eb0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857193224 0 0) 0x7f3d28004eb0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d28005940 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3d34001790 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): dump: 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053971516 0 0) 0x7f3d3c06ef30 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.862+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28004eb0 con 0x7f3d3c0b3cd0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1365806002 0 0) 0x7f3d28005940 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 >> v1:172.21.15.190:6789/0 conn(0x7f3d3c0b4820 legacy=0x7f3d3c1448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 >> v1:172.21.15.45:6789/0 conn(0x7f3d3c0b3cd0 legacy=0x7f3d3c135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 monclient: found mon.c 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c071f00 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:40:48.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 10 monclient: _renew_subs 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c1e6fd0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 10 monclient: _renew_subs 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c1e7340 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d40832640 1 librados: init done 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d34001cc0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d3959a640 4 set_mon_vals no callback set 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3d34001790 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.863+0000 7f3d237fe640 10 monclient: dump: 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.865+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f3d34019460 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d3a59c640 10 monclient: get_auth_request con 0x7f3d2806e5a0 auth_method 0 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f3d340015b0 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 10 client.4595.objecter ms_dispatch 0x7f3d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:48.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d237fe640 10 client.4595.objecter ms_handle_connect 0x7f3d2806e5a0 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d40832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d40832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d40832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.867+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c005080 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.878+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d3404e920 con 0x7f3d3c0a9ce0 2024-03-29T11:40:48.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.878+0000 7f3d237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:48.878+0000 7f3d237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:49.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.245+0000 7f3d40832640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:49.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.245+0000 7f3d40832640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:49.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.245+0000 7f3d40832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:40:49.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.245+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d1c005330 con 0x7f3d3c0a9ce0 2024-03-29T11:40:49.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.246+0000 7f3d237fe640 1 -- 172.21.15.45:0/1761849267 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f3d34050a20 con 0x7f3d3c0a9ce0 2024-03-29T11:40:49.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.246+0000 7f3d237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:49.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.246+0000 7f3d237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:49.249 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:49.249 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:49.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 10 librados: watch_flush enter 2024-03-29T11:40:49.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 10 librados: watch_flush exit 2024-03-29T11:40:49.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:40:49.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 >> v1:172.21.15.190:6816/1411598462 conn(0x7f3d2806e5a0 legacy=0x7f3d28070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 10 monclient: shutdown 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 >> v1:172.21.15.45:6790/0 conn(0x7f3d3c0a9ce0 legacy=0x7f3d3c144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 shutdown_connections 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.250+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 >> 172.21.15.45:0/1761849267 conn(0x7f3d3c06d070 msgr2=0x7f3d3c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.251+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 shutdown_connections 2024-03-29T11:40:49.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.251+0000 7f3d40832640 1 -- 172.21.15.45:0/1761849267 wait complete. 2024-03-29T11:40:49.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:49.251+0000 7f3d40832640 1 librados: shutdown 2024-03-29T11:40:49.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:56.075 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.236+0000 7f31c349a640 10 monclient: get_monmap_and_config 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.236+0000 7f31c349a640 10 monclient: build_initial_monmap 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.236+0000 7f31c349a640 10 monclient: monmap: 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:40:56.237275+0000 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:40:56.237275+0000 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.239 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.236+0000 7f31c349a640 1 Processor -- start 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.237+0000 7f31c349a640 1 -- start start 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.237+0000 7f31c349a640 10 monclient: init 2024-03-29T11:40:56.240 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc0acf60 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc13ff40 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc1403f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0b4310 con 0x7f31bc1403f0 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc140800 con 0x7f31bc0acf60 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0aec00 con 0x7f31bc13ff40 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c349a640 10 monclient(hunting): authenticate will time out at 1312.477382s 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c0a35640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f31bc13ff40 0x7f31bc148840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33292/0 (socket says 172.21.15.45:33292) 2024-03-29T11:40:56.241 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.238+0000 7f31c0a35640 1 -- 172.21.15.45:0/207236819 learned_addr learned my addr 172.21.15.45:0/207236819 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:56.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700322999 0 0) 0x7f31bc0aec00 con 0x7f31bc13ff40 2024-03-29T11:40:56.242 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a8003520 con 0x7f31bc13ff40 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741547017 0 0) 0x7f31bc0b4310 con 0x7f31bc1403f0 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31bc0aec00 con 0x7f31bc1403f0 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919060817 0 0) 0x7f31a8003520 con 0x7f31bc13ff40 2024-03-29T11:40:56.243 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.240+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 >> v1:172.21.15.190:6789/0 conn(0x7f31bc0acf60 legacy=0x7f31bc139b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 >> v1:172.21.15.45:6789/0 conn(0x7f31bc1403f0 legacy=0x7f31bc14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 1 monclient: found mon.noname-b 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:40:56.244 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0aeee0 con 0x7f31bc13ff40 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f31ac001690 con 0x7f31bc13ff40 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.241+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: dump: 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:56.245 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:56.246 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:56.246 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31c349a640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:40:56.246 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31c349a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31ac0017f0 con 0x7f31bc13ff40 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 1 -- 172.21.15.45:0/207236819 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f31ac001c40 con 0x7f31bc13ff40 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.242+0000 7f31b3fff640 10 monclient: dump: 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.248 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.243+0000 7f31c349a640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.243+0000 7f31c349a640 4 set_mon_vals no callback set 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.243+0000 7f31c349a640 10 monclient: shutdown 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 1 -- 172.21.15.45:0/207236819 >> v1:172.21.15.45:6790/0 conn(0x7f31bc13ff40 legacy=0x7f31bc148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 1 -- 172.21.15.45:0/207236819 shutdown_connections 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 1 -- 172.21.15.45:0/207236819 >> 172.21.15.45:0/207236819 conn(0x7f31bc0b35e0 msgr2=0x7f31bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 1 -- 172.21.15.45:0/207236819 shutdown_connections 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.244+0000 7f31c349a640 1 -- 172.21.15.45:0/207236819 wait complete. 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 10 monclient: build_initial_monmap 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 10 monclient: monmap: 2024-03-29T11:40:56.249 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 librados: starting msgr at 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 librados: starting objecter 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 Processor -- start 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 -- start start 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 librados: setting wanted keys 2024-03-29T11:40:56.250 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 1 librados: calling monclient init 2024-03-29T11:40:56.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.245+0000 7f31c349a640 10 monclient: init 2024-03-29T11:40:56.251 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0acf60 addr v1:172.21.15.190:6789/0 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc13ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc1403f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1d31b0 con 0x7f31bc13ff40 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.246+0000 7f31c349a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1d3d50 con 0x7f31bc0acf60 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31c349a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc1d3ee0 con 0x7f31bc1403f0 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31c349a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31c349a640 10 monclient(hunting): authenticate will time out at 1312.486128s 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31c0a35640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31bc13ff40 0x7f31bc13c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:60906/0 (socket says 172.21.15.45:60906) 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31c0a35640 1 -- 172.21.15.45:0/1220414101 learned_addr learned my addr 172.21.15.45:0/1220414101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 client.?.objecter ms_handle_connect 0x7f31bc13ff40 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:56.252 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242032943 0 0) 0x7f31bc1d31b0 con 0x7f31bc13ff40 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.247+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a0004eb0 con 0x7f31bc13ff40 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310334108 0 0) 0x7f31a0004eb0 con 0x7f31bc13ff40 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31bc1d31b0 con 0x7f31bc13ff40 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f31ac001c00 con 0x7f31bc13ff40 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.253 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2696312262 0 0) 0x7f31bc1d31b0 con 0x7f31bc13ff40 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 >> v1:172.21.15.45:6790/0 conn(0x7f31bc1403f0 legacy=0x7f31bc1dd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.248+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 >> v1:172.21.15.190:6789/0 conn(0x7f31bc0acf60 legacy=0x7f31bc1d3640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 1 monclient: found mon.a 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:56.254 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0ab350 con 0x7f31bc13ff40 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31c349a640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31c349a640 10 monclient: _renew_subs 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31c349a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:56.255 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.249+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc0ab600 con 0x7f31bc13ff40 2024-03-29T11:40:56.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31c349a640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:56.256 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31c349a640 10 monclient: _renew_subs 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31c349a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc13cb40 con 0x7f31bc13ff40 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31c349a640 1 librados: init done 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31ac001d60 con 0x7f31bc13ff40 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f31ac004cb0 con 0x7f31bc13ff40 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b1ffb640 10 monclient: dump: 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:56.257 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.250+0000 7f31b37fe640 4 set_mon_vals no callback set 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.252+0000 7f31c349a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.252+0000 7f31c349a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.252+0000 7f31c349a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.252+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c005080 con 0x7f31bc13ff40 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.252+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f31ac004e10 con 0x7f31bc13ff40 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f31ac084bd0 con 0x7f31bc13ff40 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 10 client.4527.objecter ms_dispatch 0x7f31bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:56.258 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:56.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T11:40:56.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31c1236640 10 monclient: get_auth_request con 0x7f31a006dc70 auth_method 0 2024-03-29T11:40:56.259 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.253+0000 7f31b1ffb640 10 client.4527.objecter ms_handle_connect 0x7f31a006dc70 2024-03-29T11:40:56.263 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.261+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31ac0504b0 con 0x7f31bc13ff40 2024-03-29T11:40:56.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.261+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:56.264 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.261+0000 7f31b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:56.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.590+0000 7f31c349a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.590+0000 7f31c349a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.590+0000 7f31c349a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:40:56.592 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.590+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f319c005330 con 0x7f31bc13ff40 2024-03-29T11:40:56.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.592+0000 7f31b1ffb640 1 -- 172.21.15.45:0/1220414101 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f31ac0523e0 con 0x7f31bc13ff40 2024-03-29T11:40:56.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.592+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.594 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.592+0000 7f31b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:56.595 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:40:56.595 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 10 librados: watch_flush enter 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 10 librados: watch_flush exit 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 10 client.4527.objecter successfully canceled tick 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 >> v1:172.21.15.190:6816/1411598462 conn(0x7f31a006dc70 legacy=0x7f31a0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 10 monclient: shutdown 2024-03-29T11:40:56.598 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 >> v1:172.21.15.45:6789/0 conn(0x7f31bc13ff40 legacy=0x7f31bc13c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 shutdown_connections 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.595+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 >> 172.21.15.45:0/1220414101 conn(0x7f31bc0b35e0 msgr2=0x7f31bc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.596+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 shutdown_connections 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.596+0000 7f31c349a640 1 -- 172.21.15.45:0/1220414101 wait complete. 2024-03-29T11:40:56.599 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:40:56.596+0000 7f31c349a640 1 librados: shutdown 2024-03-29T11:40:56.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:03.419 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:03.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:03.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 10 monclient: build_initial_monmap 2024-03-29T11:41:03.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 10 monclient: monmap: 2024-03-29T11:41:03.552 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:03.551307+0000 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:03.551307+0000 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:03.553 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 1 Processor -- start 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 1 -- start start 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.550+0000 7f532110e640 10 monclient: init 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.551+0000 7f532110e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f532110e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f532110e640 10 monclient(hunting): picked mon.noname-a con 0x7f531c1457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f532110e640 10 monclient(hunting): picked mon.noname-b con 0x7f531c13f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f532110e640 10 monclient(hunting): picked mon.noname-c con 0x7f531c0a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f532110e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0aaef0 con 0x7f531c1457f0 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f531a59c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f531c13f650 0x7f531c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:45432/0 (socket says 172.21.15.45:45432) 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.552+0000 7f531a59c640 1 -- 172.21.15.45:0/3999455157 learned_addr learned my addr 172.21.15.45:0/3999455157 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c145bd0 con 0x7f531c0a9d70 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c145eb0 con 0x7f531c13f650 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f532110e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f532110e640 10 monclient(hunting): authenticate will time out at 1319.792384s 2024-03-29T11:41:03.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602052807 0 0) 0x7f531c145bd0 con 0x7f531c0a9d70 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5304003520 con 0x7f531c0a9d70 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.553+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206077094 0 0) 0x7f531c145eb0 con 0x7f531c13f650 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c145bd0 con 0x7f531c13f650 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676472910 0 0) 0x7f531c145bd0 con 0x7f531c13f650 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 >> v1:172.21.15.190:6789/0 conn(0x7f531c0a9d70 legacy=0x7f531c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.554+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 >> v1:172.21.15.45:6789/0 conn(0x7f531c1457f0 legacy=0x7f531c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 1 monclient: found mon.noname-b 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c146340 con 0x7f531c13f650 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5310001670 con 0x7f531c13f650 2024-03-29T11:41:03.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: dump: 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f532110e640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:41:03.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f532110e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53100017d0 con 0x7f531c13f650 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 1 -- 172.21.15.45:0/3999455157 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5310001c20 con 0x7f531c13f650 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f5319d9b640 10 monclient: dump: 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:03.563 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f532110e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.555+0000 7f532110e640 4 set_mon_vals no callback set 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.556+0000 7f532110e640 10 monclient: shutdown 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.556+0000 7f532110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.556+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 >> v1:172.21.15.45:6790/0 conn(0x7f531c13f650 legacy=0x7f531c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.556+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 shutdown_connections 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.556+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 >> 172.21.15.45:0/3999455157 conn(0x7f531c066a00 msgr2=0x7f531c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.557+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 shutdown_connections 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.557+0000 7f532110e640 1 -- 172.21.15.45:0/3999455157 wait complete. 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.558+0000 7f532110e640 10 monclient: build_initial_monmap 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.558+0000 7f532110e640 10 monclient: monmap: 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:03.564 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.559+0000 7f532110e640 1 librados: starting msgr at 2024-03-29T11:41:03.565 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.559+0000 7f532110e640 1 librados: starting objecter 2024-03-29T11:41:03.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.559+0000 7f532110e640 1 Processor -- start 2024-03-29T11:41:03.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.560+0000 7f532110e640 1 -- start start 2024-03-29T11:41:03.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.560+0000 7f532110e640 1 librados: setting wanted keys 2024-03-29T11:41:03.569 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.560+0000 7f532110e640 1 librados: calling monclient init 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.560+0000 7f532110e640 10 monclient: init 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.561+0000 7f532110e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.561+0000 7f532110e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.562+0000 7f532110e640 10 monclient(hunting): picked mon.noname-a con 0x7f531c0a9d70 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.562+0000 7f532110e640 10 monclient(hunting): picked mon.noname-b con 0x7f531c13f650 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f531a59c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f531c13f650 0x7f531c1da4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:47446/0 (socket says 172.21.15.45:47446) 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 10 monclient(hunting): picked mon.noname-c con 0x7f531c1457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c1418c0 con 0x7f531c0a9d70 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c1421f0 con 0x7f531c13f650 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f531a59c640 1 -- 172.21.15.45:0/376866990 learned_addr learned my addr 172.21.15.45:0/376866990 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c142dc0 con 0x7f531c1457f0 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.563+0000 7f532110e640 10 monclient(hunting): authenticate will time out at 1319.803084s 2024-03-29T11:41:03.570 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 client.?.objecter ms_handle_connect 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951563623 0 0) 0x7f531c1421f0 con 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308004eb0 con 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 client.?.objecter ms_handle_connect 0x7f531c1457f0 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.564+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1258433110 0 0) 0x7f5308004eb0 con 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f531c1421f0 con 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5310001be0 con 0x7f531c13f650 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.571 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): dump: 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863696879 0 0) 0x7f531c142dc0 con 0x7f531c1457f0 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.572 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308004eb0 con 0x7f531c1457f0 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978282388 0 0) 0x7f5308004eb0 con 0x7f531c1457f0 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f531c142dc0 con 0x7f531c1457f0 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5314001cd0 con 0x7f531c1457f0 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient(hunting): dump: 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.573 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1669065446 0 0) 0x7f531c1421f0 con 0x7f531c13f650 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 >> v1:172.21.15.45:6790/0 conn(0x7f531c1457f0 legacy=0x7f531c1dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 >> v1:172.21.15.45:6789/0 conn(0x7f531c0a9d70 legacy=0x7f531c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 monclient: found mon.b 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c141af0 con 0x7f531c13f650 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:03.574 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.565+0000 7f53037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 10 monclient: _renew_subs 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c1db300 con 0x7f531c13f650 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 10 monclient: _renew_subs 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c140de0 con 0x7f531c13f650 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f532110e640 1 librados: init done 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5310004a90 con 0x7f531c13f650 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f5310004ea0 con 0x7f531c13f650 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:03.575 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f531959a640 4 set_mon_vals no callback set 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.566+0000 7f53037fe640 10 monclient: dump: 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.567+0000 7f532110e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.567+0000 7f532110e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.567+0000 7f532110e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:41:03.576 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.567+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc005080 con 0x7f531c13f650 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.569+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f531001ae70 con 0x7f531c13f650 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f5310084bf0 con 0x7f531c13f650 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f53037fe640 10 client.4498.objecter ms_dispatch 0x7f531c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f53037fe640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f53037fe640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f53037fe640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.570+0000 7f531ad9d640 10 monclient: get_auth_request con 0x7f530806de60 auth_method 0 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.571+0000 7f53037fe640 10 client.4498.objecter ms_handle_connect 0x7f530806de60 2024-03-29T11:41:03.578 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.576+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53100504d0 con 0x7f531c13f650 2024-03-29T11:41:03.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.576+0000 7f53037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.579 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.576+0000 7f53037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:03.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.893+0000 7f532110e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.893+0000 7f532110e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.893+0000 7f532110e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.893+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52fc005330 con 0x7f531c13f650 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.894+0000 7f53037fe640 1 -- 172.21.15.45:0/376866990 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f5310052400 con 0x7f531c13f650 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.894+0000 7f53037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.894+0000 7f53037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:03.897 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:03.897 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:03.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.897+0000 7f532110e640 10 librados: watch_flush enter 2024-03-29T11:41:03.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.897+0000 7f532110e640 10 librados: watch_flush exit 2024-03-29T11:41:03.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.897+0000 7f532110e640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:41:03.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.897+0000 7f532110e640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:41:03.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 >> v1:172.21.15.190:6816/1411598462 conn(0x7f530806de60 legacy=0x7f5308070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 10 monclient: shutdown 2024-03-29T11:41:03.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 >> v1:172.21.15.190:6789/0 conn(0x7f531c13f650 legacy=0x7f531c1da4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 shutdown_connections 2024-03-29T11:41:03.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 >> 172.21.15.45:0/376866990 conn(0x7f531c066a00 msgr2=0x7f531c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 shutdown_connections 2024-03-29T11:41:03.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 -- 172.21.15.45:0/376866990 wait complete. 2024-03-29T11:41:03.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:03.898+0000 7f532110e640 1 librados: shutdown 2024-03-29T11:41:03.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:10.722 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.857+0000 7feeb24e2640 10 monclient: get_monmap_and_config 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.857+0000 7feeb24e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.857+0000 7feeb24e2640 10 monclient: monmap: 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:10.858754+0000 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:10.858754+0000 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:10.860 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:10.861 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:10.861 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.858+0000 7feeb24e2640 1 Processor -- start 2024-03-29T11:41:10.862 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.858+0000 7feeb24e2640 1 -- start start 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.858+0000 7feeb24e2640 10 monclient: init 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-c con 0x7feeac0b0770 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-b con 0x7feeac0ad850 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-a con 0x7feeac13c1b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0ac8e0 con 0x7feeac13c1b0 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0ad330 con 0x7feeac0b0770 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0acfe0 con 0x7feeac0ad850 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeb24e2640 10 monclient(hunting): authenticate will time out at 1327.098521s 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeab7fe640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7feeac0ad850 0x7feeac0aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:52710/0 (socket says 172.21.15.45:52710) 2024-03-29T11:41:10.863 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeab7fe640 1 -- 172.21.15.45:0/3055885624 learned_addr learned my addr 172.21.15.45:0/3055885624 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853967621 0 0) 0x7feeac0acfe0 con 0x7feeac0ad850 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee94003520 con 0x7feeac0ad850 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757174395 0 0) 0x7fee94003520 con 0x7feeac0ad850 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 >> v1:172.21.15.190:6789/0 conn(0x7feeac0b0770 legacy=0x7feeac072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:10.864 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.859+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 >> v1:172.21.15.45:6789/0 conn(0x7feeac13c1b0 legacy=0x7feeac14a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:10.865 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 1 monclient: found mon.noname-b 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0ab1f0 con 0x7feeac0ad850 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7feea0001690 con 0x7feeac0ad850 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: dump: 2024-03-29T11:41:10.866 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeb24e2640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeb24e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea00018f0 con 0x7feeac0ad850 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.867 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 1 -- 172.21.15.45:0/3055885624 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7feea0001d40 con 0x7feeac0ad850 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeaaffd640 10 monclient: dump: 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.868 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeb24e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.862+0000 7feeb24e2640 4 set_mon_vals no callback set 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 10 monclient: shutdown 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 1 -- 172.21.15.45:0/3055885624 >> v1:172.21.15.45:6790/0 conn(0x7feeac0ad850 legacy=0x7feeac0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 1 -- 172.21.15.45:0/3055885624 shutdown_connections 2024-03-29T11:41:10.869 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 1 -- 172.21.15.45:0/3055885624 >> 172.21.15.45:0/3055885624 conn(0x7feeac06b660 msgr2=0x7feeac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 1 -- 172.21.15.45:0/3055885624 shutdown_connections 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.863+0000 7feeb24e2640 1 -- 172.21.15.45:0/3055885624 wait complete. 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.864+0000 7feeb24e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.864+0000 7feeb24e2640 10 monclient: monmap: 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:10.872 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.864+0000 7feeb24e2640 1 librados: starting msgr at 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 1 librados: starting objecter 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 1 Processor -- start 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 1 -- start start 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 1 librados: setting wanted keys 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 1 librados: calling monclient init 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.865+0000 7feeb24e2640 10 monclient: init 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-c con 0x7feeac0ad850 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:10.873 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-a con 0x7feeac0b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient(hunting): picked mon.noname-b con 0x7feeac13c1b0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac06e250 con 0x7feeac0b0770 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac070920 con 0x7feeac13c1b0 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac06e4a0 con 0x7feeac0ad850 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeb24e2640 10 monclient(hunting): authenticate will time out at 1327.105869s 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeabfff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7feeac0ad850 0x7feeac0710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53332/0 (socket says 172.21.15.45:53332) 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feeabfff640 1 -- 172.21.15.45:0/3106176830 learned_addr learned my addr 172.21.15.45:0/3106176830 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.866+0000 7feea8ff9640 10 client.?.objecter ms_handle_connect 0x7feeac0ad850 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873753476 0 0) 0x7feeac06e4a0 con 0x7feeac0ad850 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.874 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98004eb0 con 0x7feeac0ad850 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381841220 0 0) 0x7fee98004eb0 con 0x7feeac0ad850 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeac06e4a0 con 0x7feeac0ad850 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fee9c001960 con 0x7feeac0ad850 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.875 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.867+0000 7feea8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.868+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3163533138 0 0) 0x7feeac06e4a0 con 0x7feeac0ad850 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.868+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 >> v1:172.21.15.190:6789/0 conn(0x7feeac13c1b0 legacy=0x7feeac06f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.876 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.868+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 >> v1:172.21.15.45:6789/0 conn(0x7feeac0b0770 legacy=0x7feeac06e9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:10.878 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 1 monclient: found mon.c 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0717c0 con 0x7feeac0ad850 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feea8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feeb24e2640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feeb24e2640 10 monclient: _renew_subs 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feeb24e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.870+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac06d9d0 con 0x7feeac0ad850 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee9c001c80 con 0x7feeac0ad850 2024-03-29T11:41:10.879 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fee9c003a40 con 0x7feeac0ad850 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feeb24e2640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feea8ff9640 10 monclient: dump: 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:10.880 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feeb24e2640 10 monclient: _renew_subs 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feeb24e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac1e6fb0 con 0x7feeac0ad850 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.871+0000 7feeb24e2640 1 librados: init done 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.872+0000 7feeb24e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.872+0000 7feeb24e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.872+0000 7feeb24e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.872+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fee9c003ba0 con 0x7feeac0ad850 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.872+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac075f10 con 0x7feeac0ad850 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.873+0000 7feea9ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feeab7fe640 10 monclient: get_auth_request con 0x7fee9806dca0 auth_method 0 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fee9c084a30 con 0x7feeac0ad850 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 10 client.4610.objecter ms_dispatch 0x7feeac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:10.881 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:10.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:41:10.882 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.874+0000 7feea8ff9640 10 client.4610.objecter ms_handle_connect 0x7fee9806dca0 2024-03-29T11:41:10.883 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.881+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee9c050370 con 0x7feeac0ad850 2024-03-29T11:41:10.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.881+0000 7feea8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.884 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:10.881+0000 7feea8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:11.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.200+0000 7feeb24e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:11.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.200+0000 7feeb24e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:11.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.200+0000 7feeb24e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:11.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.200+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeac06e680 con 0x7feeac0ad850 2024-03-29T11:41:11.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.201+0000 7feea8ff9640 1 -- 172.21.15.45:0/3106176830 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fee9c0840e0 con 0x7feeac0ad850 2024-03-29T11:41:11.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.201+0000 7feea8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:11.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.201+0000 7feea8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:11.204 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:11.204 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 10 librados: watch_flush enter 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 10 librados: watch_flush exit 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 >> v1:172.21.15.190:6816/1411598462 conn(0x7fee9806dca0 legacy=0x7fee98070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 10 monclient: shutdown 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 >> v1:172.21.15.45:6790/0 conn(0x7feeac0ad850 legacy=0x7feeac0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 shutdown_connections 2024-03-29T11:41:11.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.204+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 >> 172.21.15.45:0/3106176830 conn(0x7feeac06b660 msgr2=0x7feeac0b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:11.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.205+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 shutdown_connections 2024-03-29T11:41:11.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.205+0000 7feeb24e2640 1 -- 172.21.15.45:0/3106176830 wait complete. 2024-03-29T11:41:11.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:11.205+0000 7feeb24e2640 1 librados: shutdown 2024-03-29T11:41:11.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:18.031 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:18.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.176+0000 7f83436d6640 10 monclient: get_monmap_and_config 2024-03-29T11:41:18.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.176+0000 7f83436d6640 10 monclient: build_initial_monmap 2024-03-29T11:41:18.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.176+0000 7f83436d6640 10 monclient: monmap: 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:18.178133+0000 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:18.178133+0000 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.178+0000 7f83436d6640 1 Processor -- start 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.178+0000 7f83436d6640 1 -- start start 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.178+0000 7f83436d6640 10 monclient: init 2024-03-29T11:41:18.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-a con 0x7f833c0ad6c0 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-c con 0x7f833c0af6c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-b con 0x7f833c14fdc0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0a94e0 con 0x7f833c0ad6c0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0af3a0 con 0x7f833c0af6c0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0afb60 con 0x7f833c14fdc0 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f83436d6640 10 monclient(hunting): authenticate will time out at 1334.418443s 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f8341c73640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f833c14fdc0 0x7f833c15a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:57678/0 (socket says 172.21.15.45:57678) 2024-03-29T11:41:18.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.179+0000 7f8341c73640 1 -- 172.21.15.45:0/2019421636 learned_addr learned my addr 172.21.15.45:0/2019421636 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:18.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466519379 0 0) 0x7f833c0afb60 con 0x7f833c14fdc0 2024-03-29T11:41:18.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:18.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:18.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:18.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:41:18.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.181+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8328003500 con 0x7f833c14fdc0 2024-03-29T11:41:18.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.182+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159114331 0 0) 0x7f8328003500 con 0x7f833c14fdc0 2024-03-29T11:41:18.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.182+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 >> v1:172.21.15.190:6789/0 conn(0x7f833c0af6c0 legacy=0x7f833c152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:18.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 >> v1:172.21.15.45:6789/0 conn(0x7f833c0ad6c0 legacy=0x7f833c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 monclient: found mon.noname-b 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0aab80 con 0x7f833c14fdc0 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f83436d6640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f83436d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8334001a10 con 0x7f833c14fdc0 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:18.187 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: dump: 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f83436d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8334001ca0 con 0x7f833c14fdc0 2024-03-29T11:41:18.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:18.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f83436d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:18.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f83436d6640 4 set_mon_vals no callback set 2024-03-29T11:41:18.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 1 -- 172.21.15.45:0/2019421636 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8334002030 con 0x7f833c14fdc0 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.184+0000 7f8333fff640 10 monclient: dump: 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.185+0000 7f83436d6640 10 monclient: shutdown 2024-03-29T11:41:18.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.185+0000 7f83436d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:18.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.185+0000 7f83436d6640 1 -- 172.21.15.45:0/2019421636 >> v1:172.21.15.45:6790/0 conn(0x7f833c14fdc0 legacy=0x7f833c15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.185+0000 7f83436d6640 1 -- 172.21.15.45:0/2019421636 shutdown_connections 2024-03-29T11:41:18.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.185+0000 7f83436d6640 1 -- 172.21.15.45:0/2019421636 >> 172.21.15.45:0/2019421636 conn(0x7f833c0b35e0 msgr2=0x7f833c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:18.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.188+0000 7f83436d6640 1 -- 172.21.15.45:0/2019421636 shutdown_connections 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.188+0000 7f83436d6640 1 -- 172.21.15.45:0/2019421636 wait complete. 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.189+0000 7f83436d6640 10 monclient: build_initial_monmap 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.189+0000 7f83436d6640 10 monclient: monmap: 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:18.192 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 librados: starting msgr at 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 librados: starting objecter 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 Processor -- start 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 -- start start 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 librados: setting wanted keys 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 1 librados: calling monclient init 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.190+0000 7f83436d6640 10 monclient: init 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.191+0000 7f83436d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:18.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.191+0000 7f83436d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:18.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-b con 0x7f833c0ad6c0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:18.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-c con 0x7f833c0af6c0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:18.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 10 monclient(hunting): picked mon.noname-a con 0x7f833c14fdc0 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:18.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0724b0 con 0x7f833c14fdc0 2024-03-29T11:41:18.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c070a10 con 0x7f833c0ad6c0 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0b1c90 con 0x7f833c0af6c0 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.192+0000 7f83436d6640 10 monclient(hunting): authenticate will time out at 1334.432067s 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.196+0000 7f8341472640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f833c0ad6c0 0x7f833c070f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:46753/0 (socket says 172.21.15.45:46753) 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.196+0000 7f8341472640 1 -- 172.21.15.45:0/2875322981 learned_addr learned my addr 172.21.15.45:0/2875322981 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.197+0000 7f8331ffb640 10 client.?.objecter ms_handle_connect 0x7f833c0ad6c0 2024-03-29T11:41:18.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.197+0000 7f8331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 client.?.objecter ms_handle_connect 0x7f833c0af6c0 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 client.?.objecter ms_handle_connect 0x7f833c14fdc0 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350789193 0 0) 0x7f833c070a10 con 0x7f833c0ad6c0 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:18.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83200092c0 con 0x7f833c0ad6c0 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552292955 0 0) 0x7f833c0b1c90 con 0x7f833c0af6c0 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c070a10 con 0x7f833c0af6c0 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919531058 0 0) 0x7f833c0724b0 con 0x7f833c14fdc0 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:41:18.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.200+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c0b1c90 con 0x7f833c14fdc0 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387128895 0 0) 0x7f833c070a10 con 0x7f833c0af6c0 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f833c0724b0 con 0x7f833c0af6c0 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f832c001790 con 0x7f833c0af6c0 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.206 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470964354 0 0) 0x7f83200092c0 con 0x7f833c0ad6c0 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f833c070a10 con 0x7f833c0ad6c0 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8338001960 con 0x7f833c0ad6c0 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:18.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1198201169 0 0) 0x7f833c0724b0 con 0x7f833c0af6c0 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 >> v1:172.21.15.190:6789/0 conn(0x7f833c0ad6c0 legacy=0x7f833c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 >> v1:172.21.15.45:6789/0 conn(0x7f833c14fdc0 legacy=0x7f833c14c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:18.208 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 monclient: found mon.c 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0b1e20 con 0x7f833c0af6c0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f8331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f83436d6640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f83436d6640 10 monclient: _renew_subs 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f83436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.201+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c1e70a0 con 0x7f833c0af6c0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.202+0000 7f83436d6640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.202+0000 7f83436d6640 10 monclient: _renew_subs 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.202+0000 7f83436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.209 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.202+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c14c9a0 con 0x7f833c0af6c0 2024-03-29T11:41:18.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.202+0000 7f83436d6640 1 librados: init done 2024-03-29T11:41:18.210 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f832c001c80 con 0x7f833c0af6c0 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f832c003830 con 0x7f833c0af6c0 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f8331ffb640 10 monclient: dump: 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:18.211 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.205+0000 7f83337fe640 4 set_mon_vals no callback set 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.206+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f832c019800 con 0x7f833c0af6c0 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.207+0000 7f83436d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.207+0000 7f83436d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.207+0000 7f83436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.212 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.207+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c005080 con 0x7f833c0af6c0 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.210+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f832c0836a0 con 0x7f833c0af6c0 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.210+0000 7f8331ffb640 10 client.4616.objecter ms_dispatch 0x7f833c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.210+0000 7f8331ffb640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.210+0000 7f8331ffb640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.210+0000 7f8331ffb640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:41:18.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.211+0000 7f8341472640 10 monclient: get_auth_request con 0x7f83200767f0 auth_method 0 2024-03-29T11:41:18.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.211+0000 7f8331ffb640 10 client.4616.objecter ms_handle_connect 0x7f83200767f0 2024-03-29T11:41:18.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.218+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f832c04f0f0 con 0x7f833c0af6c0 2024-03-29T11:41:18.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.218+0000 7f8331ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:18.221 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.218+0000 7f8331ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:18.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.539+0000 7f83436d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.539+0000 7f83436d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.539+0000 7f83436d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:18.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.539+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f831c005330 con 0x7f833c0af6c0 2024-03-29T11:41:18.544 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:18.544 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:18.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.541+0000 7f8331ffb640 1 -- 172.21.15.45:0/2875322981 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f832c051020 con 0x7f833c0af6c0 2024-03-29T11:41:18.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.541+0000 7f8331ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.541+0000 7f8331ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 10 librados: watch_flush enter 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 10 librados: watch_flush exit 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 >> v1:172.21.15.190:6816/1411598462 conn(0x7f83200767f0 legacy=0x7f8320078cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 10 monclient: shutdown 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 >> v1:172.21.15.45:6790/0 conn(0x7f833c0af6c0 legacy=0x7f833c14bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 shutdown_connections 2024-03-29T11:41:18.548 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.545+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 >> 172.21.15.45:0/2875322981 conn(0x7f833c0b35e0 msgr2=0x7f833c0a9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:18.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.546+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 shutdown_connections 2024-03-29T11:41:18.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.546+0000 7f83436d6640 1 -- 172.21.15.45:0/2875322981 wait complete. 2024-03-29T11:41:18.549 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:18.547+0000 7f83436d6640 1 librados: shutdown 2024-03-29T11:41:18.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:25.373 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.523+0000 7fc4d9601640 10 monclient: get_monmap_and_config 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.523+0000 7fc4d9601640 10 monclient: build_initial_monmap 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.523+0000 7fc4d9601640 10 monclient: monmap: 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:25.524711+0000 2024-03-29T11:41:25.526 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:25.524711+0000 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.524+0000 7fc4d9601640 1 Processor -- start 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 1 -- start start 2024-03-29T11:41:25.527 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 10 monclient: init 2024-03-29T11:41:25.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:25.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:25.530 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-c con 0x7fc4cc037220 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.525+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-a con 0x7fc4cc0cf8a0 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-b con 0x7fc4cc02b0f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc02c120 con 0x7fc4cc0cf8a0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc02c400 con 0x7fc4cc037220 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc02abd0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.526+0000 7fc4d9601640 10 monclient(hunting): authenticate will time out at 1341.765160s 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d8e00640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc4cc02b0f0 0x7fc4cc0d7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37694/0 (socket says 172.21.15.45:37694) 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d8e00640 1 -- 172.21.15.45:0/3102815153 learned_addr learned my addr 172.21.15.45:0/3102815153 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670574674 0 0) 0x7fc4cc02abd0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:25.531 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:25.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:41:25.532 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.527+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c4003520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.529+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712773818 0 0) 0x7fc4c4003520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.529+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 >> v1:172.21.15.190:6789/0 conn(0x7fc4cc037220 legacy=0x7fc4cc0cf190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.529+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 >> v1:172.21.15.45:6789/0 conn(0x7fc4cc0cf8a0 legacy=0x7fc4cc0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.529+0000 7fc4d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.529+0000 7fc4d2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0c92c0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d9601640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc4c8001690 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d9601640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: dump: 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.535 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4c8001920 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 1 -- 172.21.15.45:0/3102815153 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc4c8001cf0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:25.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d2ffd640 10 monclient: dump: 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d9601640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.530+0000 7fc4d9601640 4 set_mon_vals no callback set 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 10 monclient: shutdown 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 1 -- 172.21.15.45:0/3102815153 >> v1:172.21.15.45:6790/0 conn(0x7fc4cc02b0f0 legacy=0x7fc4cc0d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.537 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 1 -- 172.21.15.45:0/3102815153 shutdown_connections 2024-03-29T11:41:25.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 1 -- 172.21.15.45:0/3102815153 >> 172.21.15.45:0/3102815153 conn(0x7fc4cc03a4b0 msgr2=0x7fc4cc0367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 1 -- 172.21.15.45:0/3102815153 shutdown_connections 2024-03-29T11:41:25.538 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.531+0000 7fc4d9601640 1 -- 172.21.15.45:0/3102815153 wait complete. 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.532+0000 7fc4d9601640 10 monclient: build_initial_monmap 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.532+0000 7fc4d9601640 10 monclient: monmap: 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:25.542 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 librados: starting msgr at 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 librados: starting objecter 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 Processor -- start 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 -- start start 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 librados: setting wanted keys 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 1 librados: calling monclient init 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.533+0000 7fc4d9601640 10 monclient: init 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.534+0000 7fc4d9601640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.534+0000 7fc4d9601640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.534+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-c con 0x7fc4cc02b0f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.535+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-b con 0x7fc4cc037220 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.535+0000 7fc4d9601640 10 monclient(hunting): picked mon.noname-a con 0x7fc4cc168f00 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.535+0000 7fc4d9601640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc02dce0 con 0x7fc4cc168f00 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.535+0000 7fc4d9601640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc169390 con 0x7fc4cc037220 2024-03-29T11:41:25.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d3fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc4cc02b0f0 0x7fc4cc0ca5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37828/0 (socket says 172.21.15.45:37828) 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d3fff640 1 -- 172.21.15.45:0/177248522 learned_addr learned my addr 172.21.15.45:0/177248522 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 client.?.objecter ms_handle_connect 0x7fc4cc02b0f0 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4cc169520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d9601640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:25.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d9601640 10 monclient(hunting): authenticate will time out at 1341.775726s 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186499821 0 0) 0x7fc4cc169520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.536+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b8004eb0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.545 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437887150 0 0) 0x7fc4b8004eb0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4cc169520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc4d404fdd0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:25.546 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291280866 0 0) 0x7fc4cc169520 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.537+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 >> v1:172.21.15.190:6789/0 conn(0x7fc4cc037220 legacy=0x7fc4cc1687f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:25.547 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.538+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 >> v1:172.21.15.45:6789/0 conn(0x7fc4cc168f00 legacy=0x7fc4cc174bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 1 monclient: found mon.c 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc02c8e0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:25.550 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.540+0000 7fc4d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.541+0000 7fc4d9601640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.541+0000 7fc4d9601640 10 monclient: _renew_subs 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.541+0000 7fc4d9601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.541+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc02cb30 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4d404f400 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc4d4075710 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d0ff9640 10 monclient: dump: 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.551 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.542+0000 7fc4d27fc640 4 set_mon_vals no callback set 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d9601640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d9601640 10 monclient: _renew_subs 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d9601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc02d8d0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d9601640 1 librados: init done 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.543+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fc4d4067bf0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.544+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fc4d40cf5c0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.544+0000 7fc4d0ff9640 10 client.4622.objecter ms_dispatch 0x7fc4d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:25.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.545+0000 7fc4d0ff9640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.545+0000 7fc4d0ff9640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.545+0000 7fc4d0ff9640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.545+0000 7fc4d37fe640 10 monclient: get_auth_request con 0x7fc4b806db00 auth_method 0 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.546+0000 7fc4d0ff9640 10 client.4622.objecter ms_handle_connect 0x7fc4b806db00 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.546+0000 7fc4d9601640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.546+0000 7fc4d9601640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.546+0000 7fc4d9601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.546+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc004ca0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.556+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4d40cb0d0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.556+0000 7fc4d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.556+0000 7fc4d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:25.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.898+0000 7fc4d9601640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.898+0000 7fc4d9601640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.898+0000 7fc4d9601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:25.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.898+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4cc1696b0 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.900+0000 7fc4d0ff9640 1 -- 172.21.15.45:0/177248522 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fc4d40cd000 con 0x7fc4cc02b0f0 2024-03-29T11:41:25.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.900+0000 7fc4d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.900+0000 7fc4d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:25.903 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:25.903 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 10 librados: watch_flush enter 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 10 librados: watch_flush exit 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 >> v1:172.21.15.190:6816/1411598462 conn(0x7fc4b806db00 legacy=0x7fc4b806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 10 monclient: shutdown 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.907 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.904+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 >> v1:172.21.15.45:6790/0 conn(0x7fc4cc02b0f0 legacy=0x7fc4cc0ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.905+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 shutdown_connections 2024-03-29T11:41:25.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.905+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 >> 172.21.15.45:0/177248522 conn(0x7fc4cc03a4b0 msgr2=0x7fc4cc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.905+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 shutdown_connections 2024-03-29T11:41:25.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.905+0000 7fc4d9601640 1 -- 172.21.15.45:0/177248522 wait complete. 2024-03-29T11:41:25.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:25.906+0000 7fc4d9601640 1 librados: shutdown 2024-03-29T11:41:25.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:32.735 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:32.885 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.883+0000 7f1cf449a640 10 monclient: get_monmap_and_config 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.883+0000 7f1cf449a640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.883+0000 7f1cf449a640 10 monclient: monmap: 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:32.884014+0000 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:32.884014+0000 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:32.886 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.884+0000 7f1cf449a640 1 Processor -- start 2024-03-29T11:41:32.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.885+0000 7f1cf449a640 1 -- start start 2024-03-29T11:41:32.887 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.885+0000 7f1cf449a640 10 monclient: init 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec14a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec0ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0aba70 con 0x7f1cec0b0770 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0aecb0 con 0x7f1cec0ac5a0 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0af150 con 0x7f1cec14a700 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.886+0000 7f1cf449a640 10 monclient(hunting): authenticate will time out at 1349.124480s 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.887+0000 7f1cf1a35640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1cec14a700 0x7f1cec0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34782/0 (socket says 172.21.15.45:34782) 2024-03-29T11:41:32.889 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.887+0000 7f1cf1a35640 1 -- 172.21.15.45:0/3733414818 learned_addr learned my addr 172.21.15.45:0/3733414818 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252686892 0 0) 0x7f1cec0af150 con 0x7f1cec14a700 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:41:32.890 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.888+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8003520 con 0x7f1cec14a700 2024-03-29T11:41:32.892 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334628116 0 0) 0x7f1cec0aba70 con 0x7f1cec0b0770 2024-03-29T11:41:32.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:41:32.893 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.891+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec0af150 con 0x7f1cec0b0770 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926916091 0 0) 0x7f1cd8003520 con 0x7f1cec14a700 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 >> v1:172.21.15.190:6789/0 conn(0x7f1cec0ac5a0 legacy=0x7f1cec151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 >> v1:172.21.15.45:6789/0 conn(0x7f1cec0b0770 legacy=0x7f1cec149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 1 monclient: found mon.noname-b 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.894 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.892+0000 7f1cf1234640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:32.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.893+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0ac9d0 con 0x7f1cec14a700 2024-03-29T11:41:32.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.893+0000 7f1cf1234640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.893+0000 7f1cf1234640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.893+0000 7f1cf1234640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.895 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.894+0000 7f1cf449a640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:41:32.896 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.894+0000 7f1cf449a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1ce4001a10 con 0x7f1cec14a700 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: dump: 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:32.897 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:32.898 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce4001c30 con 0x7f1cec14a700 2024-03-29T11:41:32.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 1 -- 172.21.15.45:0/3733414818 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1ce4001fc0 con 0x7f1cec14a700 2024-03-29T11:41:32.899 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf1234640 10 monclient: dump: 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf449a640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.895+0000 7f1cf449a640 4 set_mon_vals no callback set 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 10 monclient: shutdown 2024-03-29T11:41:32.900 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 1 -- 172.21.15.45:0/3733414818 >> v1:172.21.15.45:6790/0 conn(0x7f1cec14a700 legacy=0x7f1cec0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 1 -- 172.21.15.45:0/3733414818 shutdown_connections 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 1 -- 172.21.15.45:0/3733414818 >> 172.21.15.45:0/3733414818 conn(0x7f1cec06b660 msgr2=0x7f1cec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 1 -- 172.21.15.45:0/3733414818 shutdown_connections 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.896+0000 7f1cf449a640 1 -- 172.21.15.45:0/3733414818 wait complete. 2024-03-29T11:41:32.901 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.898+0000 7f1cf449a640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.898+0000 7f1cf449a640 10 monclient: monmap: 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.902 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.898+0000 7f1cf449a640 1 librados: starting msgr at 2024-03-29T11:41:32.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.898+0000 7f1cf449a640 1 librados: starting objecter 2024-03-29T11:41:32.903 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.898+0000 7f1cf449a640 1 Processor -- start 2024-03-29T11:41:32.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.900+0000 7f1cf449a640 1 -- start start 2024-03-29T11:41:32.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.900+0000 7f1cf449a640 1 librados: setting wanted keys 2024-03-29T11:41:32.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.900+0000 7f1cf449a640 1 librados: calling monclient init 2024-03-29T11:41:32.904 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.900+0000 7f1cf449a640 10 monclient: init 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec0ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec0b0770 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec14a700 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec129280 con 0x7f1cec14a700 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec1479e0 con 0x7f1cec0ac5a0 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec12a0b0 con 0x7f1cec0b0770 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.901+0000 7f1cf449a640 10 monclient(hunting): authenticate will time out at 1349.139500s 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.902+0000 7f1cf1a35640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f1cec0b0770 0x7f1cec146bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:35200/0 (socket says 172.21.15.45:35200) 2024-03-29T11:41:32.905 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.902+0000 7f1cf1a35640 1 -- 172.21.15.45:0/2046494492 learned_addr learned my addr 172.21.15.45:0/2046494492 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:32.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.905+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cec0b0770 2024-03-29T11:41:32.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505092093 0 0) 0x7f1cec12a0b0 con 0x7f1cec0b0770 2024-03-29T11:41:32.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.908 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd0004eb0 con 0x7f1cec0b0770 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cec14a700 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.906+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.907+0000 7f1ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f1cec0ac5a0 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.907+0000 7f1ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.909 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.907+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101499835 0 0) 0x7f1cec129280 con 0x7f1cec14a700 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.907+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.908+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.908+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.908+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.908+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec12a0b0 con 0x7f1cec14a700 2024-03-29T11:41:32.910 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.908+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2119276775 0 0) 0x7f1cd0004eb0 con 0x7f1cec0b0770 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cec129280 con 0x7f1cec0b0770 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801288813 0 0) 0x7f1cec1479e0 con 0x7f1cec0ac5a0 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd0004eb0 con 0x7f1cec0ac5a0 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.909+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1ce4001f40 con 0x7f1cec0b0770 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:32.916 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962860327 0 0) 0x7f1cec12a0b0 con 0x7f1cec14a700 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cec1479e0 con 0x7f1cec14a700 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1cdc0017b0 con 0x7f1cec14a700 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.917 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.910+0000 7f1ce2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1206744603 0 0) 0x7f1cec1479e0 con 0x7f1cec14a700 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 >> v1:172.21.15.45:6790/0 conn(0x7f1cec0b0770 legacy=0x7f1cec146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.918 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 >> v1:172.21.15.190:6789/0 conn(0x7f1cec0ac5a0 legacy=0x7f1cec1db8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 1 monclient: found mon.a 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec12a390 con 0x7f1cec14a700 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.911+0000 7f1ce2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1ce2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 10 monclient: _renew_subs 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec1d9140 con 0x7f1cec14a700 2024-03-29T11:41:32.919 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 10 monclient: _renew_subs 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec128f40 con 0x7f1cec14a700 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1cf449a640 1 librados: init done 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cdc001ca0 con 0x7f1cec14a700 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.912+0000 7f1ce2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f1cdc003790 con 0x7f1cec14a700 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1ce2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1ce2ffd640 10 monclient: dump: 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:32.920 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1cf449a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1cf449a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1cf449a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:32.921 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.913+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec0ab560 con 0x7f1cec14a700 2024-03-29T11:41:32.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.914+0000 7f1ce3fff640 4 set_mon_vals no callback set 2024-03-29T11:41:32.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.918+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f1cdc0038f0 con 0x7f1cec14a700 2024-03-29T11:41:32.922 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.919+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f1cdc083740 con 0x7f1cec14a700 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.919+0000 7f1cf2236640 10 monclient: get_auth_request con 0x7f1cd006df90 auth_method 0 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.919+0000 7f1ce2ffd640 10 client.4536.objecter ms_dispatch 0x7f1cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.920+0000 7f1ce2ffd640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.920+0000 7f1ce2ffd640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.920+0000 7f1ce2ffd640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:41:32.923 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.921+0000 7f1ce2ffd640 10 client.4536.objecter ms_handle_connect 0x7f1cd006df90 2024-03-29T11:41:32.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.926+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cdc04ee50 con 0x7f1cec14a700 2024-03-29T11:41:32.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.926+0000 7f1ce2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:32.926+0000 7f1ce2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:33.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.300+0000 7f1cf449a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:33.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.300+0000 7f1cf449a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:33.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.300+0000 7f1cf449a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:33.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.300+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cec075f10 con 0x7f1cec14a700 2024-03-29T11:41:33.305 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:33.305 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:33.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.303+0000 7f1ce2ffd640 1 -- 172.21.15.45:0/2046494492 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f1cdc050f50 con 0x7f1cec14a700 2024-03-29T11:41:33.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.303+0000 7f1ce2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:33.305 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.303+0000 7f1ce2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:33.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.306+0000 7f1cf449a640 10 librados: watch_flush enter 2024-03-29T11:41:33.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 10 librados: watch_flush exit 2024-03-29T11:41:33.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:41:33.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 >> v1:172.21.15.190:6816/1411598462 conn(0x7f1cd006df90 legacy=0x7f1cd0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 10 monclient: shutdown 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.307+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 >> v1:172.21.15.45:6789/0 conn(0x7f1cec14a700 legacy=0x7f1cec1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.308+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 shutdown_connections 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.308+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 >> 172.21.15.45:0/2046494492 conn(0x7f1cec06b660 msgr2=0x7f1cec147c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.308+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 shutdown_connections 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.308+0000 7f1cf449a640 1 -- 172.21.15.45:0/2046494492 wait complete. 2024-03-29T11:41:33.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:33.308+0000 7f1cf449a640 1 librados: shutdown 2024-03-29T11:41:33.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:40.136 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:40.283 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.281+0000 7f4bbc910640 10 monclient: get_monmap_and_config 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.281+0000 7f4bbc910640 10 monclient: build_initial_monmap 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.281+0000 7f4bbc910640 10 monclient: monmap: 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:40.281911+0000 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:40.281911+0000 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:40.284 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:40.285 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:40.285 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.285 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.282+0000 7f4bbc910640 1 Processor -- start 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.283+0000 7f4bbc910640 1 -- start start 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.283+0000 7f4bbc910640 10 monclient: init 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb81457f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80aaef0 con 0x7f4bb813f650 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb8145bd0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb8145eb0 con 0x7f4bb81457f0 2024-03-29T11:41:40.286 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:40.287 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.284+0000 7f4bbc910640 10 monclient(hunting): authenticate will time out at 1356.522531s 2024-03-29T11:41:40.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.287+0000 7f4bb659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4bb81457f0 0x7f4bb813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33492/0 (socket says 172.21.15.45:33492) 2024-03-29T11:41:40.289 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.287+0000 7f4bb659c640 1 -- 172.21.15.45:0/3607959031 learned_addr learned my addr 172.21.15.45:0/3607959031 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:40.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091823122 0 0) 0x7f4bb8145eb0 con 0x7f4bb81457f0 2024-03-29T11:41:40.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:40.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:40.290 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:40.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:41:40.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.288+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba0003520 con 0x7f4bb81457f0 2024-03-29T11:41:40.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147356496 0 0) 0x7f4bb80aaef0 con 0x7f4bb813f650 2024-03-29T11:41:40.291 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb8145eb0 con 0x7f4bb813f650 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650795021 0 0) 0x7f4ba0003520 con 0x7f4bb81457f0 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 >> v1:172.21.15.190:6789/0 conn(0x7f4bb80a9d70 legacy=0x7f4bb814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:40.292 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 >> v1:172.21.15.45:6789/0 conn(0x7f4bb813f650 legacy=0x7f4bb80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 monclient: found mon.noname-b 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8146340 con 0x7f4bb81457f0 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: _finish_auth 0 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4ba8001690 con 0x7f4bb81457f0 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:40.293 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bb559a640 10 monclient: dump: 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:40.294 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:40.295 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bbc910640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:41:40.295 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.289+0000 7f4bbc910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.290+0000 7f4bb559a640 1 -- 172.21.15.45:0/3607959031 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ba80017f0 con 0x7f4bb81457f0 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.290+0000 7f4bb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.290+0000 7f4bbc910640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.290+0000 7f4bbc910640 4 set_mon_vals no callback set 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.291+0000 7f4bbc910640 10 monclient: shutdown 2024-03-29T11:41:40.296 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.291+0000 7f4bbc910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:40.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.291+0000 7f4bbc910640 1 -- 172.21.15.45:0/3607959031 >> v1:172.21.15.45:6790/0 conn(0x7f4bb81457f0 legacy=0x7f4bb813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.291+0000 7f4bbc910640 1 -- 172.21.15.45:0/3607959031 shutdown_connections 2024-03-29T11:41:40.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.291+0000 7f4bbc910640 1 -- 172.21.15.45:0/3607959031 >> 172.21.15.45:0/3607959031 conn(0x7f4bb8066a00 msgr2=0x7f4bb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:40.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.292+0000 7f4bbc910640 1 -- 172.21.15.45:0/3607959031 shutdown_connections 2024-03-29T11:41:40.297 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.293+0000 7f4bbc910640 1 -- 172.21.15.45:0/3607959031 wait complete. 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.294+0000 7f4bbc910640 10 monclient: build_initial_monmap 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.294+0000 7f4bbc910640 10 monclient: monmap: 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:40.298 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:40.299 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:40.299 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.294+0000 7f4bbc910640 1 librados: starting msgr at 2024-03-29T11:41:40.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.294+0000 7f4bbc910640 1 librados: starting objecter 2024-03-29T11:41:40.299 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.295+0000 7f4bbc910640 1 Processor -- start 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.295+0000 7f4bbc910640 1 -- start start 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.295+0000 7f4bbc910640 1 librados: setting wanted keys 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.295+0000 7f4bbc910640 1 librados: calling monclient init 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.295+0000 7f4bbc910640 10 monclient: init 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:40.300 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb81457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb806ffa0 con 0x7f4bb813f650 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb8070280 con 0x7f4bb81457f0 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb81e7680 con 0x7f4bb80a9d70 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.296+0000 7f4bbc910640 10 monclient(hunting): authenticate will time out at 1356.534660s 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.298+0000 7f4bb659c640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f4bb80a9d70 0x7f4bb81e7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:37976/0 (socket says 172.21.15.45:37976) 2024-03-29T11:41:40.301 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.298+0000 7f4bb659c640 1 -- 172.21.15.45:0/786942928 learned_addr learned my addr 172.21.15.45:0/786942928 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:40.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.299+0000 7f4ba6ffd640 10 client.?.objecter ms_handle_connect 0x7f4bb80a9d70 2024-03-29T11:41:40.302 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.299+0000 7f4ba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.300+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695379500 0 0) 0x7f4bb81e7680 con 0x7f4bb80a9d70 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.300+0000 7f4ba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.300+0000 7f4ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.300+0000 7f4ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.300+0000 7f4ba6ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:41:40.303 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.301+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b98004eb0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.301+0000 7f4ba6ffd640 10 client.?.objecter ms_handle_connect 0x7f4bb813f650 2024-03-29T11:41:40.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.301+0000 7f4ba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:40.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.301+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841865757 0 0) 0x7f4bb806ffa0 con 0x7f4bb813f650 2024-03-29T11:41:40.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.301+0000 7f4ba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:40.306 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb81e7680 con 0x7f4bb813f650 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276931366 0 0) 0x7f4b98004eb0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bb806ffa0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4ba8004980 con 0x7f4bb80a9d70 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:40.307 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.302+0000 7f4ba6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988591011 0 0) 0x7f4bb81e7680 con 0x7f4bb813f650 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b98004eb0 con 0x7f4bb813f650 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4bac001970 con 0x7f4bb813f650 2024-03-29T11:41:40.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:40.309 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4131470844 0 0) 0x7f4bb806ffa0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.303+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 >> v1:172.21.15.190:6789/0 conn(0x7f4bb81457f0 legacy=0x7f4bb81ef920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 >> v1:172.21.15.45:6789/0 conn(0x7f4bb813f650 legacy=0x7f4bb81e6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 1 monclient: found mon.c 2024-03-29T11:41:40.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb806de90 con 0x7f4bb80a9d70 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4ba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4bbc910640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4bbc910640 10 monclient: _renew_subs 2024-03-29T11:41:40.314 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4bbc910640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.304+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb8070a60 con 0x7f4bb80a9d70 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bbc910640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bbc910640 10 monclient: _renew_subs 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bbc910640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb81e8490 con 0x7f4bb80a9d70 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bbc910640 1 librados: init done 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ba8001a60 con 0x7f4bb80a9d70 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f4ba8004f30 con 0x7f4bb80a9d70 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:40.315 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4ba6ffd640 10 monclient: dump: 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.305+0000 7f4bb4d99640 4 set_mon_vals no callback set 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.307+0000 7f4bbc910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:40.316 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.308+0000 7f4bbc910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.308+0000 7f4bbc910640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.308+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c005080 con 0x7f4bb80a9d70 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.309+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f4ba801af00 con 0x7f4bb80a9d70 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f4ba8084ae0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4bb5d9b640 10 monclient: get_auth_request con 0x7f4b9806de70 auth_method 0 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 10 client.4634.objecter ms_dispatch 0x7f4bb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:41:40.317 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.310+0000 7f4ba6ffd640 10 client.4634.objecter ms_handle_connect 0x7f4b9806de70 2024-03-29T11:41:40.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.321+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ba80503c0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.321+0000 7f4ba6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:40.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.321+0000 7f4ba6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:40.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.667+0000 7f4bbc910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.667+0000 7f4bbc910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.667+0000 7f4bbc910640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.667+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b9c005330 con 0x7f4bb80a9d70 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.668+0000 7f4ba6ffd640 1 -- 172.21.15.45:0/786942928 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f4ba80522f0 con 0x7f4bb80a9d70 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.669+0000 7f4ba6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.670 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.669+0000 7f4ba6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:40.671 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:40.671 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 10 librados: watch_flush enter 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 10 librados: watch_flush exit 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 >> v1:172.21.15.190:6816/1411598462 conn(0x7f4b9806de70 legacy=0x7f4b98070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 10 monclient: shutdown 2024-03-29T11:41:40.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 >> v1:172.21.15.45:6790/0 conn(0x7f4bb80a9d70 legacy=0x7f4bb81e7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 shutdown_connections 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.677+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 >> 172.21.15.45:0/786942928 conn(0x7f4bb8066a00 msgr2=0x7f4bb813abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.678+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 shutdown_connections 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.678+0000 7f4bbc910640 1 -- 172.21.15.45:0/786942928 wait complete. 2024-03-29T11:41:40.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:40.678+0000 7f4bbc910640 1 librados: shutdown 2024-03-29T11:41:40.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:47.503 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:47.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.639+0000 7f275453a640 10 monclient: get_monmap_and_config 2024-03-29T11:41:47.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.639+0000 7f275453a640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.642 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.639+0000 7f275453a640 10 monclient: monmap: 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:47.641080+0000 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:47.641080+0000 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:47.643 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:47.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.641+0000 7f275453a640 1 Processor -- start 2024-03-29T11:41:47.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.641+0000 7f275453a640 1 -- start start 2024-03-29T11:41:47.644 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.641+0000 7f275453a640 10 monclient: init 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.642+0000 7f275453a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.642+0000 7f275453a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.642+0000 7f275453a640 10 monclient(hunting): picked mon.noname-c con 0x7f274c06c250 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.642+0000 7f275453a640 10 monclient(hunting): picked mon.noname-b con 0x7f274c072400 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 10 monclient(hunting): picked mon.noname-a con 0x7f274c06c630 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c06d760 con 0x7f274c06c630 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c070500 con 0x7f274c06c250 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0aff50 con 0x7f274c072400 2024-03-29T11:41:47.647 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f275453a640 10 monclient(hunting): authenticate will time out at 1363.882151s 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f2752ad7640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f274c06c630 0x7f274c151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:41290/0 (socket says 172.21.15.45:41290) 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f2752ad7640 1 -- 172.21.15.45:0/751193461 learned_addr learned my addr 172.21.15.45:0/751193461 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94685753 0 0) 0x7f274c06d760 con 0x7f274c06c630 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2738003520 con 0x7f274c06c630 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.643+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2506876679 0 0) 0x7f2738003520 con 0x7f274c06c630 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.644+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 >> v1:172.21.15.45:6790/0 conn(0x7f274c072400 legacy=0x7f274c1465b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:47.648 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.644+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 >> v1:172.21.15.190:6789/0 conn(0x7f274c06c250 legacy=0x7f274c071cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 1 monclient: found mon.noname-a 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0b00e0 con 0x7f274c06c630 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f273c001690 con 0x7f274c06c630 2024-03-29T11:41:47.651 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f27512d4640 10 monclient: dump: 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f275453a640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:41:47.652 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.645+0000 7f275453a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.646+0000 7f27512d4640 1 -- 172.21.15.45:0/751193461 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f273c001860 con 0x7f274c06c630 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.646+0000 7f27512d4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.646+0000 7f275453a640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.646+0000 7f275453a640 4 set_mon_vals no callback set 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 10 monclient: shutdown 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 1 -- 172.21.15.45:0/751193461 >> v1:172.21.15.45:6789/0 conn(0x7f274c06c630 legacy=0x7f274c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 1 -- 172.21.15.45:0/751193461 shutdown_connections 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 1 -- 172.21.15.45:0/751193461 >> 172.21.15.45:0/751193461 conn(0x7f274c067610 msgr2=0x7f274c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 1 -- 172.21.15.45:0/751193461 shutdown_connections 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.647+0000 7f275453a640 1 -- 172.21.15.45:0/751193461 wait complete. 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.648+0000 7f275453a640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.648+0000 7f275453a640 10 monclient: monmap: 2024-03-29T11:41:47.653 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:47.654 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 librados: starting msgr at 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 librados: starting objecter 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 Processor -- start 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 -- start start 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 librados: setting wanted keys 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 1 librados: calling monclient init 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.649+0000 7f275453a640 10 monclient: init 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.650+0000 7f275453a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.650+0000 7f275453a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:47.661 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 10 monclient(hunting): picked mon.noname-b con 0x7f274c06c250 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 10 monclient(hunting): picked mon.noname-c con 0x7f274c06c630 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f2751ad5640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f274c06c630 0x7f274c1423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:40068/0 (socket says 172.21.15.45:40068) 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f2751ad5640 1 -- 172.21.15.45:0/1328211089 learned_addr learned my addr 172.21.15.45:0/1328211089 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 10 monclient(hunting): picked mon.noname-a con 0x7f274c072400 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f2742ffd640 10 client.?.objecter ms_handle_connect 0x7f274c06c630 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f2742ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0a9660 con 0x7f274c072400 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0b5950 con 0x7f274c06c250 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.651+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0aa690 con 0x7f274c06c630 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f275453a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f275453a640 10 monclient(hunting): authenticate will time out at 1363.891207s 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926206134 0 0) 0x7f274c0aa690 con 0x7f274c06c630 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.662 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2730004eb0 con 0x7f274c06c630 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 client.?.objecter ms_handle_connect 0x7f274c072400 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825693810 0 0) 0x7f274c0a9660 con 0x7f274c072400 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.652+0000 7f2742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f274c0aa690 con 0x7f274c072400 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3727831735 0 0) 0x7f2730004eb0 con 0x7f274c06c630 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f274c0a9660 con 0x7f274c06c630 2024-03-29T11:41:47.663 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f2744001ad0 con 0x7f274c06c630 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:47.664 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2030536762 0 0) 0x7f274c0aa690 con 0x7f274c072400 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.653+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2730004eb0 con 0x7f274c072400 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.654+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 178921744 0 0) 0x7f274c0a9660 con 0x7f274c06c630 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.654+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 >> v1:172.21.15.190:6789/0 conn(0x7f274c06c250 legacy=0x7f274c0a9b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.655+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 >> v1:172.21.15.45:6789/0 conn(0x7f274c072400 legacy=0x7f274c1e7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 1 monclient: found mon.c 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0b3610 con 0x7f274c06c630 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2744001c30 con 0x7f274c06c630 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f275453a640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:41:47.665 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f275453a640 10 monclient: _renew_subs 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f275453a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.656+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c0aa290 con 0x7f274c06c630 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f275453a640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f275453a640 10 monclient: _renew_subs 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f275453a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c1dc600 con 0x7f274c06c630 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f275453a640 1 librados: init done 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2742ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f27440020e0 con 0x7f274c06c630 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2742ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2742ffd640 10 monclient: dump: 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:47.666 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:47.667 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.657+0000 7f2750ad3640 4 set_mon_vals no callback set 2024-03-29T11:41:47.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.659+0000 7f275453a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.668 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.659+0000 7f275453a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.659+0000 7f275453a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.659+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2734005080 con 0x7f274c06c630 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.661+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f274401abf0 con 0x7f274c06c630 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f2744084ca0 con 0x7f274c06c630 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 10 client.4637.objecter ms_dispatch 0x7f274c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f27522d6640 10 monclient: get_auth_request con 0x7f273006dfb0 auth_method 0 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:41:47.669 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.662+0000 7f2742ffd640 10 client.4637.objecter ms_handle_connect 0x7f273006dfb0 2024-03-29T11:41:47.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.669+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27440503b0 con 0x7f274c06c630 2024-03-29T11:41:47.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.669+0000 7f2742ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.672 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.669+0000 7f2742ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:48.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.999+0000 7f275453a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.999+0000 7f275453a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.999+0000 7f275453a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:41:48.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:47.999+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2734005330 con 0x7f274c06c630 2024-03-29T11:41:48.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.000+0000 7f2742ffd640 1 -- 172.21.15.45:0/1328211089 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f27440524b0 con 0x7f274c06c630 2024-03-29T11:41:48.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.000+0000 7f2742ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:48.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.000+0000 7f2742ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:48.003 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:48.003 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:48.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.004+0000 7f275453a640 10 librados: watch_flush enter 2024-03-29T11:41:48.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.004+0000 7f275453a640 10 librados: watch_flush exit 2024-03-29T11:41:48.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.004+0000 7f275453a640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:41:48.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.004+0000 7f275453a640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.007+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 >> v1:172.21.15.190:6816/1411598462 conn(0x7f273006dfb0 legacy=0x7f2730070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.007+0000 7f275453a640 10 monclient: shutdown 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.007+0000 7f275453a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.007+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 >> v1:172.21.15.45:6790/0 conn(0x7f274c06c630 legacy=0x7f274c1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.008+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 shutdown_connections 2024-03-29T11:41:48.010 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.008+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 >> 172.21.15.45:0/1328211089 conn(0x7f274c067610 msgr2=0x7f274c14a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:48.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.010+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 shutdown_connections 2024-03-29T11:41:48.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.014+0000 7f275453a640 1 -- 172.21.15.45:0/1328211089 wait complete. 2024-03-29T11:41:48.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:48.014+0000 7f275453a640 1 librados: shutdown 2024-03-29T11:41:48.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:54.845 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:54.985 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.982+0000 7f47f16ae640 10 monclient: get_monmap_and_config 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.982+0000 7f47f16ae640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.982+0000 7f47f16ae640 10 monclient: monmap: 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:41:54.984063+0000 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:41:54.984063+0000 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:41:54.986 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:41:54.987 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.983+0000 7f47f16ae640 1 Processor -- start 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.983+0000 7f47f16ae640 1 -- start start 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.983+0000 7f47f16ae640 10 monclient: init 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-b con 0x7f47ec071820 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-a con 0x7f47ec0af660 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-c con 0x7f47ec06ab70 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec0729c0 con 0x7f47ec0af660 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec0afaf0 con 0x7f47ec06ab70 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec0b1110 con 0x7f47ec071820 2024-03-29T11:41:54.988 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47f16ae640 10 monclient(hunting): authenticate will time out at 1371.223702s 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47eaffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f47ec071820 0x7f47ec142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:43174/0 (socket says 172.21.15.45:43174) 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.984+0000 7f47eaffd640 1 -- 172.21.15.45:0/3437750101 learned_addr learned my addr 172.21.15.45:0/3437750101 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318360511 0 0) 0x7f47ec0b1110 con 0x7f47ec071820 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d4003520 con 0x7f47ec071820 2024-03-29T11:41:54.989 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2566890639 0 0) 0x7f47d4003520 con 0x7f47ec071820 2024-03-29T11:41:54.990 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.985+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 >> v1:172.21.15.190:6789/0 conn(0x7f47ec06ab70 legacy=0x7f47ec14d930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 >> v1:172.21.15.45:6789/0 conn(0x7f47ec0af660 legacy=0x7f47ec072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec0b12a0 con 0x7f47ec071820 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.993 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f47dc001690 con 0x7f47ec071820 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: dump: 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:54.994 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47f16ae640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47f16ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47dc0017f0 con 0x7f47ec071820 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 1 -- 172.21.15.45:0/3437750101 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f47dc001c40 con 0x7f47ec071820 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47e9ffb640 10 monclient: dump: 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:54.995 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47f16ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.988+0000 7f47f16ae640 4 set_mon_vals no callback set 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.989+0000 7f47f16ae640 10 monclient: shutdown 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.989+0000 7f47f16ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.996 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.989+0000 7f47f16ae640 1 -- 172.21.15.45:0/3437750101 >> v1:172.21.15.45:6790/0 conn(0x7f47ec071820 legacy=0x7f47ec142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.999 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.991+0000 7f47f16ae640 1 -- 172.21.15.45:0/3437750101 shutdown_connections 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.991+0000 7f47f16ae640 1 -- 172.21.15.45:0/3437750101 >> 172.21.15.45:0/3437750101 conn(0x7f47ec0b4930 msgr2=0x7f47ec0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.991+0000 7f47f16ae640 1 -- 172.21.15.45:0/3437750101 shutdown_connections 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.991+0000 7f47f16ae640 1 -- 172.21.15.45:0/3437750101 wait complete. 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.992+0000 7f47f16ae640 10 monclient: build_initial_monmap 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.992+0000 7f47f16ae640 10 monclient: monmap: 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:41:55.000 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.992+0000 7f47f16ae640 1 librados: starting msgr at 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.992+0000 7f47f16ae640 1 librados: starting objecter 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.993+0000 7f47f16ae640 1 Processor -- start 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.993+0000 7f47f16ae640 1 -- start start 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.993+0000 7f47f16ae640 1 librados: setting wanted keys 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.993+0000 7f47f16ae640 1 librados: calling monclient init 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.993+0000 7f47f16ae640 10 monclient: init 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47f16ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47f16ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-c con 0x7f47ec06ab70 addr v1:172.21.15.45:6790/0 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47eaffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f47ec06ab70 0x7f47ec1da900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:44040/0 (socket says 172.21.15.45:44040) 2024-03-29T11:41:55.001 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47eaffd640 1 -- 172.21.15.45:0/1747071595 learned_addr learned my addr 172.21.15.45:0/1747071595 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47d37fe640 10 client.?.objecter ms_handle_connect 0x7f47ec06ab70 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.994+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-a con 0x7f47ec071820 addr v1:172.21.15.45:6789/0 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 client.?.objecter ms_handle_connect 0x7f47ec071820 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 10 monclient(hunting): picked mon.noname-b con 0x7f47ec0af660 addr v1:172.21.15.190:6789/0 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec14ba70 con 0x7f47ec071820 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec1db3d0 con 0x7f47ec0af660 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ec1db010 con 0x7f47ec06ab70 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47f16ae640 10 monclient(hunting): authenticate will time out at 1371.235232s 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798027903 0 0) 0x7f47ec14ba70 con 0x7f47ec071820 2024-03-29T11:41:55.002 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8004eb0 con 0x7f47ec071820 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121922522 0 0) 0x7f47ec1db010 con 0x7f47ec06ab70 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ec14ba70 con 0x7f47ec06ab70 2024-03-29T11:41:55.003 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1739559953 0 0) 0x7f47d8004eb0 con 0x7f47ec071820 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47ec1db010 con 0x7f47ec071820 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f47e00018f0 con 0x7f47ec071820 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): dump: 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:55.004 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.996+0000 7f47d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864005571 0 0) 0x7f47ec14ba70 con 0x7f47ec06ab70 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47d8004eb0 con 0x7f47ec06ab70 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4120504494 0 0) 0x7f47ec1db010 con 0x7f47ec071820 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 >> v1:172.21.15.45:6790/0 conn(0x7f47ec06ab70 legacy=0x7f47ec1da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 >> v1:172.21.15.190:6789/0 conn(0x7f47ec0af660 legacy=0x7f47ec149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 monclient: found mon.a 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec1db1a0 con 0x7f47ec071820 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:55.005 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 10 monclient: _renew_subs 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec14e510 con 0x7f47ec071820 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 10 monclient: _renew_subs 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec14e860 con 0x7f47ec071820 2024-03-29T11:41:55.006 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.997+0000 7f47f16ae640 1 librados: init done 2024-03-29T11:41:55.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e0003610 con 0x7f47ec071820 2024-03-29T11:41:55.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:55.007 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f47e0003920 con 0x7f47ec071820 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47d37fe640 10 monclient: dump: 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:41:55.008 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.998+0000 7f47e97fa640 4 set_mon_vals no callback set 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:54.999+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f47e00198f0 con 0x7f47ec071820 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7f47e0083840 con 0x7f47ec071820 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 10 client.4554.objecter ms_dispatch 0x7f47ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47eaffd640 10 monclient: get_auth_request con 0x7f47d806dd30 auth_method 0 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47d37fe640 10 client.4554.objecter ms_handle_connect 0x7f47d806dd30 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47f16ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47f16ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47f16ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:55.009 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.000+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc005080 con 0x7f47ec071820 2024-03-29T11:41:55.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.008+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47e004f120 con 0x7f47ec071820 2024-03-29T11:41:55.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.008+0000 7f47d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:55.011 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.008+0000 7f47d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.334+0000 7f47f16ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.334+0000 7f47f16ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.334+0000 7f47f16ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.334+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47cc005330 con 0x7f47ec071820 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.335+0000 7f47d37fe640 1 -- 172.21.15.45:0/1747071595 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7f47e0051050 con 0x7f47ec071820 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.335+0000 7f47d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:55.339 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.335+0000 7f47d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:55.340 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:41:55.340 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 10 librados: watch_flush enter 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 10 librados: watch_flush exit 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 >> v1:172.21.15.190:6816/1411598462 conn(0x7f47d806dd30 legacy=0x7f47d80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 10 monclient: shutdown 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 >> v1:172.21.15.45:6789/0 conn(0x7f47ec071820 legacy=0x7f47ec13eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:55.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 shutdown_connections 2024-03-29T11:41:55.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 >> 172.21.15.45:0/1747071595 conn(0x7f47ec0b4930 msgr2=0x7f47ec071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:55.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 shutdown_connections 2024-03-29T11:41:55.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.338+0000 7f47f16ae640 1 -- 172.21.15.45:0/1747071595 wait complete. 2024-03-29T11:41:55.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:41:55.339+0000 7f47f16ae640 1 librados: shutdown 2024-03-29T11:41:55.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:02.163 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:02.308 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.305+0000 7fc5605f6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.305+0000 7fc5605f6640 10 monclient: build_initial_monmap 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.305+0000 7fc5605f6640 10 monclient: monmap: 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:02.307090+0000 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:02.307090+0000 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:02.309 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.306+0000 7fc5605f6640 1 Processor -- start 2024-03-29T11:42:02.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.306+0000 7fc5605f6640 1 -- start start 2024-03-29T11:42:02.310 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.306+0000 7fc5605f6640 10 monclient: init 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.307+0000 7fc5605f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.307+0000 7fc5605f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.307+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc5581422b0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc558142690 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc5580b17d0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc558142c80 con 0x7fc558142690 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580aae40 con 0x7fc5581422b0 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55813fbd0 con 0x7fc5580b17d0 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:02.311 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc5605f6640 10 monclient(hunting): authenticate will time out at 1378.547176s 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc55db91640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc558142690 0x7fc5580b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:51574/0 (socket says 172.21.15.45:51574) 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.308+0000 7fc55db91640 1 -- 172.21.15.45:0/3487143181 learned_addr learned my addr 172.21.15.45:0/3487143181 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420885701 0 0) 0x7fc558142c80 con 0x7fc558142690 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _negotiate 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc544003520 con 0x7fc558142690 2024-03-29T11:42:02.321 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13119507 0 0) 0x7fc5580aae40 con 0x7fc5581422b0 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _negotiate 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc558142c80 con 0x7fc5581422b0 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55e392640 1 -- 172.21.15.45:0/3487143181 >> v1:172.21.15.190:6789/0 conn(0x7fc5581422b0 legacy=0x7fc55813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55e392640 1 -- 172.21.15.45:0/3487143181 >> v1:172.21.15.190:6789/0 conn(0x7fc5581422b0 legacy=0x7fc55813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55e392640 1 --1- 172.21.15.45:0/3487143181 >> v1:172.21.15.190:6789/0 conn(0x7fc5581422b0 0x7fc55813f4c0 :-1 s=OPENED pgs=1187952 cs=1 l=1).handle_message read tag failed 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55e392640 1 --1- 172.21.15.45:0/3487143181 >> v1:172.21.15.190:6789/0 conn(0x7fc5581422b0 0x7fc55813f4c0 :-1 s=OPENED pgs=1187952 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.190:6789/0 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212961538 0 0) 0x7fc544003520 con 0x7fc558142690 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 >> v1:172.21.15.45:6790/0 conn(0x7fc5580b17d0 legacy=0x7fc5580b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 >> v1:172.21.15.190:6789/0 conn(0x7fc5581422b0 legacy=0x7fc55813f4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:42:02.322 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: _finish_hunting 0 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 monclient: found mon.noname-a 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558151940 con 0x7fc558142690 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: _finish_auth 0 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: _check_auth_tickets 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc5605f6640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc5605f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc550001690 con 0x7fc558142690 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc55d390640 10 monclient: dump: 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:02.323 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.309+0000 7fc5605f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5500018b0 con 0x7fc558142690 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc5605f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:02.324 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc5605f6640 4 set_mon_vals no callback set 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 1 -- 172.21.15.45:0/3487143181 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc550001bc0 con 0x7fc558142690 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc55d390640 10 monclient: dump: 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.325 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc5605f6640 10 monclient: shutdown 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc5605f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.310+0000 7fc5605f6640 1 -- 172.21.15.45:0/3487143181 >> v1:172.21.15.45:6789/0 conn(0x7fc558142690 legacy=0x7fc5580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.311+0000 7fc5605f6640 1 -- 172.21.15.45:0/3487143181 shutdown_connections 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.311+0000 7fc5605f6640 1 -- 172.21.15.45:0/3487143181 >> 172.21.15.45:0/3487143181 conn(0x7fc558066a00 msgr2=0x7fc558066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.311+0000 7fc5605f6640 1 -- 172.21.15.45:0/3487143181 shutdown_connections 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.311+0000 7fc5605f6640 1 -- 172.21.15.45:0/3487143181 wait complete. 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 10 monclient: build_initial_monmap 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 10 monclient: monmap: 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:02.326 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 librados: starting msgr at 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 librados: starting objecter 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 Processor -- start 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 -- start start 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 librados: setting wanted keys 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 1 librados: calling monclient init 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.312+0000 7fc5605f6640 10 monclient: init 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:02.327 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc5580b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc5581422b0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc558142690 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580ac280 con 0x7fc558142690 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55813c750 con 0x7fc5580b17d0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580abd30 con 0x7fc5581422b0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc5605f6640 10 monclient(hunting): authenticate will time out at 1378.552391s 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc55db91640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fc5581422b0 0x7fc55813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:33650/0 (socket says 172.21.15.45:33650) 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc55db91640 1 -- 172.21.15.45:0/662514658 learned_addr learned my addr 172.21.15.45:0/662514658 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc54effd640 10 client.?.objecter ms_handle_connect 0x7fc558142690 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc54effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc54effd640 10 client.?.objecter ms_handle_connect 0x7fc5581422b0 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.313+0000 7fc54effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:02.328 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186069340 0 0) 0x7fc5580ac280 con 0x7fc558142690 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c004eb0 con 0x7fc558142690 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 client.?.objecter ms_handle_connect 0x7fc5580b17d0 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220827444 0 0) 0x7fc53c004eb0 con 0x7fc558142690 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5580ac280 con 0x7fc558142690 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc548001840 con 0x7fc558142690 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): dump: 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:02.329 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259581246 0 0) 0x7fc5580abd30 con 0x7fc5581422b0 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:02.330 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c004eb0 con 0x7fc5581422b0 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1480955888 0 0) 0x7fc5580ac280 con 0x7fc558142690 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 >> v1:172.21.15.45:6790/0 conn(0x7fc5581422b0 legacy=0x7fc55813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 >> v1:172.21.15.190:6789/0 conn(0x7fc5580b17d0 legacy=0x7fc55806f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 monclient: found mon.a 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580acaf0 con 0x7fc558142690 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient: _finish_auth 0 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc54effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 10 monclient: _renew_subs 2024-03-29T11:42:02.331 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580acdd0 con 0x7fc558142690 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 10 monclient: _renew_subs 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5580ad390 con 0x7fc558142690 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.314+0000 7fc5605f6640 1 librados: init done 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5480035f0 con 0x7fc558142690 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fc548003a40 con 0x7fc558142690 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc54effd640 10 monclient: dump: 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:02.332 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.315+0000 7fc55cb8f640 4 set_mon_vals no callback set 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.316+0000 7fc5605f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.317+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fc548003ba0 con 0x7fc558142690 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.317+0000 7fc5605f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.317+0000 7fc5605f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fc5480837d0 con 0x7fc558142690 2024-03-29T11:42:02.333 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 10 client.4560.objecter ms_dispatch 0x7fc558000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.317+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc540005080 con 0x7fc558142690 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc55e392640 10 monclient: get_auth_request con 0x7fc53c06ddc0 auth_method 0 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.318+0000 7fc54effd640 10 client.4560.objecter ms_handle_connect 0x7fc53c06ddc0 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.326+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc54804f0b0 con 0x7fc558142690 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.326+0000 7fc54effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:02.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.326+0000 7fc54effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:02.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.697+0000 7fc5605f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.697+0000 7fc5605f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.697+0000 7fc5605f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.697+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc540005330 con 0x7fc558142690 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.698+0000 7fc54effd640 1 -- 172.21.15.45:0/662514658 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fc548050fe0 con 0x7fc558142690 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.698+0000 7fc54effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.698+0000 7fc54effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:02.702 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:02.702 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:02.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 10 librados: watch_flush enter 2024-03-29T11:42:02.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 10 librados: watch_flush exit 2024-03-29T11:42:02.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:42:02.705 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 >> v1:172.21.15.190:6816/1411598462 conn(0x7fc53c06ddc0 legacy=0x7fc53c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 10 monclient: shutdown 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.702+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 >> v1:172.21.15.45:6789/0 conn(0x7fc558142690 legacy=0x7fc55813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.703+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 shutdown_connections 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.703+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 >> 172.21.15.45:0/662514658 conn(0x7fc558066a00 msgr2=0x7fc55813e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.703+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 shutdown_connections 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.703+0000 7fc5605f6640 1 -- 172.21.15.45:0/662514658 wait complete. 2024-03-29T11:42:02.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:02.703+0000 7fc5605f6640 1 librados: shutdown 2024-03-29T11:42:02.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:09.527 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:09.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.673+0000 7fd227154640 10 monclient: get_monmap_and_config 2024-03-29T11:42:09.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.673+0000 7fd227154640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.676 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.673+0000 7fd227154640 10 monclient: monmap: 2024-03-29T11:42:09.676 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:09.676 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:09.674705+0000 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:09.674705+0000 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:09.677 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.674+0000 7fd227154640 1 Processor -- start 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.674+0000 7fd227154640 1 -- start start 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.674+0000 7fd227154640 10 monclient: init 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient(hunting): picked mon.noname-a con 0x7fd2200b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient(hunting): picked mon.noname-c con 0x7fd22014a700 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient(hunting): picked mon.noname-b con 0x7fd2200ac5a0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2200aba70 con 0x7fd2200b0770 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2200aecb0 con 0x7fd22014a700 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2200af150 con 0x7fd2200ac5a0 2024-03-29T11:42:09.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.675+0000 7fd227154640 10 monclient(hunting): authenticate will time out at 1385.914698s 2024-03-29T11:42:09.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.678+0000 7fd224ef0640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd2200b0770 0x7fd220149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:47869/0 (socket says 172.21.15.45:47869) 2024-03-29T11:42:09.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.678+0000 7fd2256f1640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7fd2200ac5a0 0x7fd220151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47452/0 (socket says 172.21.15.45:47452) 2024-03-29T11:42:09.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.678+0000 7fd21ffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd22014a700 0x7fd2200abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:34365/0 (socket says 172.21.15.45:34365) 2024-03-29T11:42:09.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.678+0000 7fd21ffff640 1 -- 172.21.15.45:0/2753192613 learned_addr learned my addr 172.21.15.45:0/2753192613 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595003120 0 0) 0x7fd2200af150 con 0x7fd2200ac5a0 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.679+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd208003520 con 0x7fd2200ac5a0 2024-03-29T11:42:09.684 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087525057 0 0) 0x7fd2200aba70 con 0x7fd2200b0770 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2200af150 con 0x7fd2200b0770 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442108534 0 0) 0x7fd208003520 con 0x7fd2200ac5a0 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 >> v1:172.21.15.190:6789/0 conn(0x7fd22014a700 legacy=0x7fd2200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 >> v1:172.21.15.45:6789/0 conn(0x7fd2200b0770 legacy=0x7fd220149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2200ac9d0 con 0x7fd2200ac5a0 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd218001990 con 0x7fd2200ac5a0 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.680+0000 7fd21f7fe640 10 monclient: dump: 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:09.686 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd227154640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd227154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd21f7fe640 1 -- 172.21.15.45:0/2753192613 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd218001af0 con 0x7fd2200ac5a0 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd21f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd227154640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:09.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.681+0000 7fd227154640 4 set_mon_vals no callback set 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.682+0000 7fd227154640 10 monclient: shutdown 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.682+0000 7fd227154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.682+0000 7fd227154640 1 -- 172.21.15.45:0/2753192613 >> v1:172.21.15.45:6790/0 conn(0x7fd2200ac5a0 legacy=0x7fd220151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.682+0000 7fd227154640 1 -- 172.21.15.45:0/2753192613 shutdown_connections 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.682+0000 7fd227154640 1 -- 172.21.15.45:0/2753192613 >> 172.21.15.45:0/2753192613 conn(0x7fd22006b660 msgr2=0x7fd22006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.685+0000 7fd227154640 1 -- 172.21.15.45:0/2753192613 shutdown_connections 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.685+0000 7fd227154640 1 -- 172.21.15.45:0/2753192613 wait complete. 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 10 monclient: monmap: 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:09.689 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 librados: starting msgr at 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 librados: starting objecter 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 Processor -- start 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 -- start start 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 librados: setting wanted keys 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 1 librados: calling monclient init 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.686+0000 7fd227154640 10 monclient: init 2024-03-29T11:42:09.690 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.687+0000 7fd227154640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.691 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.687+0000 7fd227154640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.687+0000 7fd227154640 10 monclient(hunting): picked mon.noname-b con 0x7fd2200ac5a0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 10 monclient(hunting): picked mon.noname-a con 0x7fd2200b0770 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 10 monclient(hunting): picked mon.noname-c con 0x7fd22014a700 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22012a170 con 0x7fd2200b0770 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd224ef0640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd2200ac5a0 0x7fd220128c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:34433/0 (socket says 172.21.15.45:34433) 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd224ef0640 1 -- 172.21.15.45:0/2408645946 learned_addr learned my addr 172.21.15.45:0/2408645946 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2201287b0 con 0x7fd2200ac5a0 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd220129340 con 0x7fd22014a700 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd227154640 10 monclient(hunting): authenticate will time out at 1385.930806s 2024-03-29T11:42:09.696 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd21d7fa640 10 client.?.objecter ms_handle_connect 0x7fd2200ac5a0 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.691+0000 7fd21d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 client.?.objecter ms_handle_connect 0x7fd22014a700 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 client.?.objecter ms_handle_connect 0x7fd2200b0770 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300214908 0 0) 0x7fd2201287b0 con 0x7fd2200ac5a0 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd20c004eb0 con 0x7fd2200ac5a0 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312874684 0 0) 0x7fd22012a170 con 0x7fd2200b0770 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2201287b0 con 0x7fd2200b0770 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350009664 0 0) 0x7fd220129340 con 0x7fd22014a700 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.692+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22012a170 con 0x7fd22014a700 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933234874 0 0) 0x7fd2201287b0 con 0x7fd2200b0770 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd220129340 con 0x7fd2200b0770 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd210001780 con 0x7fd2200b0770 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:09.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:09.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051932531 0 0) 0x7fd22012a170 con 0x7fd22014a700 2024-03-29T11:42:09.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2201287b0 con 0x7fd22014a700 2024-03-29T11:42:09.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd218004d00 con 0x7fd22014a700 2024-03-29T11:42:09.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.706 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.693+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446735795 0 0) 0x7fd220129340 con 0x7fd2200b0770 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 >> v1:172.21.15.45:6790/0 conn(0x7fd22014a700 legacy=0x7fd2201db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 >> v1:172.21.15.190:6789/0 conn(0x7fd2200ac5a0 legacy=0x7fd220128c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.707 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 1 monclient: found mon.a 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd220147b90 con 0x7fd2200b0770 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd210001b90 con 0x7fd2200b0770 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 10 monclient: _renew_subs 2024-03-29T11:42:09.708 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd220127d10 con 0x7fd2200b0770 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 10 monclient: _renew_subs 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.694+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd220147870 con 0x7fd2200b0770 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd227154640 1 librados: init done 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd21d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fd210001d80 con 0x7fd2200b0770 2024-03-29T11:42:09.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd21d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd21d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.695+0000 7fd21e7fc640 4 set_mon_vals no callback set 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.696+0000 7fd21d7fa640 10 monclient: dump: 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:09.710 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.696+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fd210004980 con 0x7fd2200b0770 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.697+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1355919991 0 0) 0x7fd210084800 con 0x7fd2200b0770 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.697+0000 7fd224ef0640 10 monclient: get_auth_request con 0x7fd20c06dfd0 auth_method 0 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.697+0000 7fd21d7fa640 10 client.4566.objecter ms_dispatch 0x7fd220000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd21d7fa640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd21d7fa640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd21d7fa640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd21d7fa640 10 client.4566.objecter ms_handle_connect 0x7fd20c06dfd0 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd227154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd227154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd227154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:09.711 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.698+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd200005080 con 0x7fd2200b0770 2024-03-29T11:42:09.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.706+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd21004ff10 con 0x7fd2200b0770 2024-03-29T11:42:09.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.706+0000 7fd21d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.712 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:09.706+0000 7fd21d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:10.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.038+0000 7fd227154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:10.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.038+0000 7fd227154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:10.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.038+0000 7fd227154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.45:6789/0 2024-03-29T11:42:10.042 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.038+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 --> v1:172.21.15.45:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd200005330 con 0x7fd2200b0770 2024-03-29T11:42:10.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.042+0000 7fd21d7fa640 1 -- 172.21.15.45:0/2408645946 <== mon.0 v1:172.21.15.45:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2101118874) 0x7fd210052010 con 0x7fd2200b0770 2024-03-29T11:42:10.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.042+0000 7fd21d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:10.046 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:10.046 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":10,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:32:13.144260+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:32:13.144260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:10.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.042+0000 7fd21d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:10.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.045+0000 7fd227154640 10 librados: watch_flush enter 2024-03-29T11:42:10.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.046+0000 7fd227154640 10 librados: watch_flush exit 2024-03-29T11:42:10.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.046+0000 7fd227154640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T11:42:10.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.046+0000 7fd227154640 10 client.4566.objecter successfully canceled tick 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 >> v1:172.21.15.190:6816/1411598462 conn(0x7fd20c06dfd0 legacy=0x7fd20c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 10 monclient: shutdown 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 >> v1:172.21.15.45:6789/0 conn(0x7fd2200b0770 legacy=0x7fd220146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 shutdown_connections 2024-03-29T11:42:10.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 >> 172.21.15.45:0/2408645946 conn(0x7fd22006b660 msgr2=0x7fd22006d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:10.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 shutdown_connections 2024-03-29T11:42:10.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 -- 172.21.15.45:0/2408645946 wait complete. 2024-03-29T11:42:10.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:10.047+0000 7fd227154640 1 librados: shutdown 2024-03-29T11:42:10.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:16.878 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:17.012 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.009+0000 7f6f11815640 10 monclient: get_monmap_and_config 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.009+0000 7f6f11815640 10 monclient: build_initial_monmap 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.009+0000 7f6f11815640 10 monclient: monmap: 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:17.010789+0000 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:17.010789+0000 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:17.013 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:17.014 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.011+0000 7f6f11815640 1 Processor -- start 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.011+0000 7f6f11815640 1 -- start start 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.011+0000 7f6f11815640 10 monclient: init 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0c06cac0 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c06cea0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0c145e80 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c0ac850 con 0x7f6f0c06cac0 2024-03-29T11:42:17.015 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c070470 con 0x7f6f0c145e80 2024-03-29T11:42:17.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c072600 con 0x7f6f0c06cea0 2024-03-29T11:42:17.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:17.016 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f11815640 10 monclient(hunting): authenticate will time out at 1393.251390s 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f0a7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6f0c06cea0 0x7f6f0c145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41282/0 (socket says 172.21.15.45:41282) 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f0a7fc640 1 -- 172.21.15.45:0/2998669254 learned_addr learned my addr 172.21.15.45:0/2998669254 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285421510 0 0) 0x7f6f0c072600 con 0x7f6f0c06cea0 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.012+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4003520 con 0x7f6f0c06cea0 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.013+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906208158 0 0) 0x7f6ef4003520 con 0x7f6f0c06cea0 2024-03-29T11:42:17.017 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.013+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 >> v1:172.21.15.190:6789/0 conn(0x7f6f0c145e80 legacy=0x7f6f0c14db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:17.018 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.013+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 >> v1:172.21.15.45:6789/0 conn(0x7f6f0c06cac0 legacy=0x7f6f0c071ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:17.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:17.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 1 monclient: found mon.noname-b 2024-03-29T11:42:17.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:17.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.020 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c1462d0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f11815640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc001690 con 0x7f6f0c06cea0 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f11815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:17.021 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f09ffb640 10 monclient: dump: 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.015+0000 7f6f11815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:17.022 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc001860 con 0x7f6f0c06cea0 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 1 -- 172.21.15.45:0/2998669254 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc001c70 con 0x7f6f0c06cea0 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f09ffb640 10 monclient: dump: 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.023 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f11815640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.016+0000 7f6f11815640 4 set_mon_vals no callback set 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.017+0000 7f6f11815640 10 monclient: shutdown 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.017+0000 7f6f11815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.017+0000 7f6f11815640 1 -- 172.21.15.45:0/2998669254 >> v1:172.21.15.45:6790/0 conn(0x7f6f0c06cea0 legacy=0x7f6f0c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.017+0000 7f6f11815640 1 -- 172.21.15.45:0/2998669254 shutdown_connections 2024-03-29T11:42:17.024 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.017+0000 7f6f11815640 1 -- 172.21.15.45:0/2998669254 >> 172.21.15.45:0/2998669254 conn(0x7f6f0c0675b0 msgr2=0x7f6f0c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.018+0000 7f6f11815640 1 -- 172.21.15.45:0/2998669254 shutdown_connections 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.019+0000 7f6f11815640 1 -- 172.21.15.45:0/2998669254 wait complete. 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.020+0000 7f6f11815640 10 monclient: build_initial_monmap 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.020+0000 7f6f11815640 10 monclient: monmap: 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:17.029 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.020+0000 7f6f11815640 1 librados: starting msgr at 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.020+0000 7f6f11815640 1 librados: starting objecter 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.020+0000 7f6f11815640 1 Processor -- start 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.021+0000 7f6f11815640 1 -- start start 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.021+0000 7f6f11815640 1 librados: setting wanted keys 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.021+0000 7f6f11815640 1 librados: calling monclient init 2024-03-29T11:42:17.030 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.021+0000 7f6f11815640 10 monclient: init 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.022+0000 7f6f11815640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.022+0000 7f6f11815640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.022+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0c06cac0 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.022+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0c06cea0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c145e80 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c1d4160 con 0x7f6f0c06cac0 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c13f930 con 0x7f6f0c145e80 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c13fac0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f11815640 10 monclient(hunting): authenticate will time out at 1393.262139s 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f0a7fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f6f0c06cea0 0x7f6f0c1dad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:58950/0 (socket says 172.21.15.45:58950) 2024-03-29T11:42:17.031 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6f0a7fc640 1 -- 172.21.15.45:0/577580138 learned_addr learned my addr 172.21.15.45:0/577580138 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 client.?.objecter ms_handle_connect 0x7f6f0c06cea0 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859469473 0 0) 0x7f6f0c1462d0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef8004eb0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931034100 0 0) 0x7f6ef8004eb0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f0c1462d0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.032 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc0019d0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): dump: 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:17.033 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.023+0000 7f6ef3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:17.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.024+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459549581 0 0) 0x7f6f0c1462d0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.024+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 >> v1:172.21.15.190:6789/0 conn(0x7f6f0c145e80 legacy=0x7f6f0c1db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:17.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.024+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 >> v1:172.21.15.45:6789/0 conn(0x7f6f0c06cac0 legacy=0x7f6f0c1d4680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:17.034 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 1 monclient: found mon.c 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c1d4d90 con 0x7f6f0c06cea0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6ef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 10 monclient: _renew_subs 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c1d4fe0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:17.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 10 monclient: _renew_subs 2024-03-29T11:42:17.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c1e3210 con 0x7f6f0c06cea0 2024-03-29T11:42:17.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.025+0000 7f6f11815640 1 librados: init done 2024-03-29T11:42:17.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.026+0000 7f6f11815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:17.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.026+0000 7f6f11815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:17.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.026+0000 7f6f11815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.026+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c1d4350 con 0x7f6f0c06cea0 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.027+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc001bf0 con 0x7f6f0c06cea0 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.027+0000 7f6ef3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.027+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f6efc004d90 con 0x7f6f0c06cea0 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.027+0000 7f6f097fa640 4 set_mon_vals no callback set 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 10 monclient: dump: 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:17.038 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f6efc01ad60 con 0x7f6f0c06cea0 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6f0affd640 10 monclient: get_auth_request con 0x7f6ef806db50 auth_method 0 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7f6efc084c20 con 0x7f6f0c06cea0 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 10 client.4661.objecter ms_dispatch 0x7f6f0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:17.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.028+0000 7f6ef3fff640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:17.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.029+0000 7f6ef3fff640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:42:17.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.029+0000 7f6ef3fff640 10 client.4661.objecter ms_handle_connect 0x7f6ef806db50 2024-03-29T11:42:17.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.036+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6efc050330 con 0x7f6f0c06cea0 2024-03-29T11:42:17.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.036+0000 7f6ef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:17.040 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.036+0000 7f6ef3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:17.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.348+0000 7f6f11815640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:17.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.348+0000 7f6f11815640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:17.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.348+0000 7f6f11815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:17.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.348+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f0c075f10 con 0x7f6f0c06cea0 2024-03-29T11:42:17.351 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.349+0000 7f6ef3fff640 1 -- 172.21.15.45:0/577580138 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7f6efc052430 con 0x7f6f0c06cea0 2024-03-29T11:42:17.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.349+0000 7f6ef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:17.352 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.349+0000 7f6ef3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:17.352 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:17.352 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:17.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.352+0000 7f6f11815640 10 librados: watch_flush enter 2024-03-29T11:42:17.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.352+0000 7f6f11815640 10 librados: watch_flush exit 2024-03-29T11:42:17.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.352+0000 7f6f11815640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:42:17.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.352+0000 7f6f11815640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:42:17.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.353+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 >> v1:172.21.15.190:6816/1411598462 conn(0x7f6ef806db50 legacy=0x7f6ef8070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:17.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.353+0000 7f6f11815640 10 monclient: shutdown 2024-03-29T11:42:17.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.353+0000 7f6f11815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:17.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.353+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 >> v1:172.21.15.45:6790/0 conn(0x7f6f0c06cea0 legacy=0x7f6f0c1dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:17.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.354+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 shutdown_connections 2024-03-29T11:42:17.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.354+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 >> 172.21.15.45:0/577580138 conn(0x7f6f0c0675b0 msgr2=0x7f6f0c0b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:17.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.354+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 shutdown_connections 2024-03-29T11:42:17.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.354+0000 7f6f11815640 1 -- 172.21.15.45:0/577580138 wait complete. 2024-03-29T11:42:17.358 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:17.355+0000 7f6f11815640 1 librados: shutdown 2024-03-29T11:42:17.374 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:24.181 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:24.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.328+0000 7f9cc937b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:24.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.328+0000 7f9cc937b640 10 monclient: build_initial_monmap 2024-03-29T11:42:24.332 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.328+0000 7f9cc937b640 10 monclient: monmap: 2024-03-29T11:42:24.332 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:24.332 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:24.329603+0000 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:24.329603+0000 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:24.333 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.330+0000 7f9cc937b640 1 Processor -- start 2024-03-29T11:42:24.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.330+0000 7f9cc937b640 1 -- start start 2024-03-29T11:42:24.334 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.330+0000 7f9cc937b640 10 monclient: init 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-b con 0x7f9cc40b4100 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-c con 0x7f9cc413ee90 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-a con 0x7f9cc40af900 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:24.335 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc40afd10 con 0x7f9cc40af900 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc40af6f0 con 0x7f9cc413ee90 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc40aaf60 con 0x7f9cc40b4100 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.331+0000 7f9cc937b640 10 monclient(hunting): authenticate will time out at 1400.570817s 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.332+0000 7f9cc2ffd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9cc40b4100 0x7f9cc40a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:41082/0 (socket says 172.21.15.45:41082) 2024-03-29T11:42:24.336 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.332+0000 7f9cc2ffd640 1 -- 172.21.15.45:0/2385889744 learned_addr learned my addr 172.21.15.45:0/2385889744 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233591809 0 0) 0x7f9cc40aaf60 con 0x7f9cc40b4100 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cac003520 con 0x7f9cc40b4100 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431277109 0 0) 0x7f9cac003520 con 0x7f9cc40b4100 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.333+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 >> v1:172.21.15.190:6789/0 conn(0x7f9cc413ee90 legacy=0x7f9cc40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:24.337 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.334+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 >> v1:172.21.15.45:6789/0 conn(0x7f9cc40af900 legacy=0x7f9cc413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc413e970 con 0x7f9cc40b4100 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9cb4001670 con 0x7f9cc40b4100 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:24.342 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.337+0000 7f9cc1ffb640 10 monclient: dump: 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.343 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.339+0000 7f9cc937b640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.339+0000 7f9cc937b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.340+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cb40018d0 con 0x7f9cc40b4100 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.340+0000 7f9cc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.340+0000 7f9cc937b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.340+0000 7f9cc937b640 4 set_mon_vals no callback set 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc1ffb640 1 -- 172.21.15.45:0/2385889744 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9cb4001d20 con 0x7f9cc40b4100 2024-03-29T11:42:24.344 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc1ffb640 10 monclient: dump: 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:24.345 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc937b640 10 monclient: shutdown 2024-03-29T11:42:24.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc937b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:24.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc937b640 1 -- 172.21.15.45:0/2385889744 >> v1:172.21.15.45:6790/0 conn(0x7f9cc40b4100 legacy=0x7f9cc40a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc937b640 1 -- 172.21.15.45:0/2385889744 shutdown_connections 2024-03-29T11:42:24.346 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.341+0000 7f9cc937b640 1 -- 172.21.15.45:0/2385889744 >> 172.21.15.45:0/2385889744 conn(0x7f9cc40b35e0 msgr2=0x7f9cc40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:24.347 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.344+0000 7f9cc937b640 1 -- 172.21.15.45:0/2385889744 shutdown_connections 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.344+0000 7f9cc937b640 1 -- 172.21.15.45:0/2385889744 wait complete. 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.345+0000 7f9cc937b640 10 monclient: build_initial_monmap 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.345+0000 7f9cc937b640 10 monclient: monmap: 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:24.348 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.345+0000 7f9cc937b640 1 librados: starting msgr at 2024-03-29T11:42:24.349 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.345+0000 7f9cc937b640 1 librados: starting objecter 2024-03-29T11:42:24.350 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.346+0000 7f9cc937b640 1 Processor -- start 2024-03-29T11:42:24.353 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.346+0000 7f9cc937b640 1 -- start start 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.346+0000 7f9cc937b640 1 librados: setting wanted keys 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.346+0000 7f9cc937b640 1 librados: calling monclient init 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.346+0000 7f9cc937b640 10 monclient: init 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-b con 0x7f9cc40af900 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-c con 0x7f9cc413ee90 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient(hunting): picked mon.noname-a con 0x7f9cc4131960 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:24.354 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc406f910 con 0x7f9cc4131960 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc4126d20 con 0x7f9cc40af900 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc406f150 con 0x7f9cc413ee90 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc937b640 10 monclient(hunting): authenticate will time out at 1400.586347s 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc27fc640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f9cc413ee90 0x7f9cc4125f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:48762/0 (socket says 172.21.15.45:48762) 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.347+0000 7f9cc27fc640 1 -- 172.21.15.45:0/1009595156 learned_addr learned my addr 172.21.15.45:0/1009595156 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.348+0000 7f9cab7fe640 10 client.?.objecter ms_handle_connect 0x7f9cc413ee90 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.348+0000 7f9cab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259841071 0 0) 0x7f9cc406f150 con 0x7f9cc413ee90 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:24.355 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.349+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb0004eb0 con 0x7f9cc413ee90 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430253988 0 0) 0x7f9cb0004eb0 con 0x7f9cc413ee90 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cc406f150 con 0x7f9cc413ee90 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9cbc001b50 con 0x7f9cc413ee90 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.356 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1225270590 0 0) 0x7f9cc406f150 con 0x7f9cc413ee90 2024-03-29T11:42:24.357 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.350+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 >> v1:172.21.15.190:6789/0 conn(0x7f9cc40af900 legacy=0x7f9cc406ea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:24.360 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 >> v1:172.21.15.45:6789/0 conn(0x7f9cc4131960 legacy=0x7f9cc4126610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 1 monclient: found mon.c 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc4070440 con 0x7f9cc413ee90 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.352+0000 7f9cab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 10 monclient: _renew_subs 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc4070720 con 0x7f9cc413ee90 2024-03-29T11:42:24.361 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 10 monclient: _renew_subs 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc4070ce0 con 0x7f9cc413ee90 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.353+0000 7f9cc937b640 1 librados: init done 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cbc001cb0 con 0x7f9cc413ee90 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f9cbc004dc0 con 0x7f9cc413ee90 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cab7fe640 10 monclient: dump: 2024-03-29T11:42:24.362 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.354+0000 7f9cc0ff9640 4 set_mon_vals no callback set 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.355+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f9cbc01ae50 con 0x7f9cc413ee90 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.355+0000 7f9cc937b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7f9cbc084e30 con 0x7f9cc413ee90 2024-03-29T11:42:24.363 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cab7fe640 10 client.4667.objecter ms_dispatch 0x7f9cc4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cab7fe640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cab7fe640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cab7fe640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cc937b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cc937b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca4005080 con 0x7f9cc413ee90 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.356+0000 7f9cc2ffd640 10 monclient: get_auth_request con 0x7f9cb006dc00 auth_method 0 2024-03-29T11:42:24.364 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.357+0000 7f9cab7fe640 10 client.4667.objecter ms_handle_connect 0x7f9cb006dc00 2024-03-29T11:42:24.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.364+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9cbc050730 con 0x7f9cc413ee90 2024-03-29T11:42:24.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.364+0000 7f9cab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:24.367 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.364+0000 7f9cab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:24.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.689+0000 7f9cc937b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.689+0000 7f9cc937b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.689+0000 7f9cc937b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:24.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.689+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca4005330 con 0x7f9cc413ee90 2024-03-29T11:42:24.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.691+0000 7f9cab7fe640 1 -- 172.21.15.45:0/1009595156 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7f9cbc052660 con 0x7f9cc413ee90 2024-03-29T11:42:24.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.691+0000 7f9cab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.691+0000 7f9cab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:24.694 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:24.695 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:24.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.694+0000 7f9cc937b640 10 librados: watch_flush enter 2024-03-29T11:42:24.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.694+0000 7f9cc937b640 10 librados: watch_flush exit 2024-03-29T11:42:24.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.694+0000 7f9cc937b640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:42:24.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.694+0000 7f9cc937b640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:42:24.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 >> v1:172.21.15.190:6816/1411598462 conn(0x7f9cb006dc00 legacy=0x7f9cb00700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 10 monclient: shutdown 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 >> v1:172.21.15.45:6790/0 conn(0x7f9cc413ee90 legacy=0x7f9cc4125f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 shutdown_connections 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 >> 172.21.15.45:0/1009595156 conn(0x7f9cc40b35e0 msgr2=0x7f9cc40b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 shutdown_connections 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 -- 172.21.15.45:0/1009595156 wait complete. 2024-03-29T11:42:24.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:24.695+0000 7f9cc937b640 1 librados: shutdown 2024-03-29T11:42:24.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:31.523 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:31.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.669+0000 7f8870132640 10 monclient: get_monmap_and_config 2024-03-29T11:42:31.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.669+0000 7f8870132640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.673 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.669+0000 7f8870132640 10 monclient: monmap: 2024-03-29T11:42:31.673 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:31.673 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:31.670918+0000 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:31.670918+0000 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:31.674 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:31.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.672+0000 7f8870132640 1 Processor -- start 2024-03-29T11:42:31.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.672+0000 7f8870132640 1 -- start start 2024-03-29T11:42:31.678 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.672+0000 7f8870132640 10 monclient: init 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.673+0000 7f8870132640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.673+0000 7f8870132640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.673+0000 7f8870132640 10 monclient(hunting): picked mon.noname-a con 0x7f88680a9930 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.673+0000 7f8870132640 10 monclient(hunting): picked mon.noname-c con 0x7f8868149dc0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886e6cf640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f886814ff40 0x7f8868152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:34360/0 (socket says 172.21.15.45:34360) 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886e6cf640 1 -- 172.21.15.45:0/2710976346 learned_addr learned my addr 172.21.15.45:0/2710976346 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 10 monclient(hunting): picked mon.noname-b con 0x7f886814ff40 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88680aae60 con 0x7f88680a9930 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88680b4310 con 0x7f8868149dc0 2024-03-29T11:42:31.679 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88680afdd0 con 0x7f886814ff40 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f8870132640 10 monclient(hunting): authenticate will time out at 1407.913635s 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459026372 0 0) 0x7f88680afdd0 con 0x7f886814ff40 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.674+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8854003520 con 0x7f886814ff40 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.675+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3149143672 0 0) 0x7f8854003520 con 0x7f886814ff40 2024-03-29T11:42:31.680 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.675+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 >> v1:172.21.15.190:6789/0 conn(0x7f8868149dc0 legacy=0x7f886814a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:31.681 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.677+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 >> v1:172.21.15.45:6789/0 conn(0x7f88680a9930 legacy=0x7f8868143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 1 monclient: found mon.noname-b 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886814a950 con 0x7f886814ff40 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8858001690 con 0x7f886814ff40 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.682 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.678+0000 7f886cecc640 10 monclient: dump: 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f8870132640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f8870132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8858001920 con 0x7f886814ff40 2024-03-29T11:42:31.683 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f886cecc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f8870132640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.679+0000 7f8870132640 4 set_mon_vals no callback set 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 10 monclient: shutdown 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 >> v1:172.21.15.45:6790/0 conn(0x7f886814ff40 legacy=0x7f8868152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f886cecc640 1 -- 172.21.15.45:0/2710976346 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8858001cb0 con 0x7f886814ff40 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f886cecc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 shutdown_connections 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 >> 172.21.15.45:0/2710976346 conn(0x7f88680b35e0 msgr2=0x7f88680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 shutdown_connections 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.680+0000 7f8870132640 1 -- 172.21.15.45:0/2710976346 wait complete. 2024-03-29T11:42:31.685 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.681+0000 7f8870132640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.681+0000 7f8870132640 10 monclient: monmap: 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:31.686 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:31.687 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:31.687 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:31.687 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.682+0000 7f8870132640 1 librados: starting msgr at 2024-03-29T11:42:31.687 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.682+0000 7f8870132640 1 librados: starting objecter 2024-03-29T11:42:31.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.682+0000 7f8870132640 1 Processor -- start 2024-03-29T11:42:31.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.682+0000 7f8870132640 1 -- start start 2024-03-29T11:42:31.688 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.682+0000 7f8870132640 1 librados: setting wanted keys 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.683+0000 7f8870132640 1 librados: calling monclient init 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.683+0000 7f8870132640 10 monclient: init 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.683+0000 7f8870132640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.684+0000 7f8870132640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.684+0000 7f8870132640 10 monclient(hunting): picked mon.noname-b con 0x7f88680a9930 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 10 monclient(hunting): picked mon.noname-c con 0x7f8868149dc0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 10 monclient(hunting): picked mon.noname-a con 0x7f886814ff40 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f886d6cd640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f8868149dc0 0x7f88681dbb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:36130/0 (socket says 172.21.15.45:36130) 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f886d6cd640 1 -- 172.21.15.45:0/3624345515 learned_addr learned my addr 172.21.15.45:0/3624345515 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88681db380 con 0x7f886814ff40 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8868147430 con 0x7f88680a9930 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88681db620 con 0x7f8868149dc0 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.689 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.685+0000 7f8870132640 10 monclient(hunting): authenticate will time out at 1407.924677s 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 client.?.objecter ms_handle_connect 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597292872 0 0) 0x7f88681db620 con 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f884c004eb0 con 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206113200 0 0) 0x7f884c004eb0 con 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88681db620 con 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8864001870 con 0x7f8868149dc0 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.692 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:31.693 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.688+0000 7f885e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.689+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2836757583 0 0) 0x7f88681db620 con 0x7f8868149dc0 2024-03-29T11:42:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.689+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 >> v1:172.21.15.190:6789/0 conn(0x7f88680a9930 legacy=0x7f8868146110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:31.694 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.690+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 >> v1:172.21.15.45:6789/0 conn(0x7f886814ff40 legacy=0x7f88681e5dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 1 monclient: found mon.c 2024-03-29T11:42:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:31.697 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88681dc230 con 0x7f8868149dc0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 10 monclient: _renew_subs 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8868145390 con 0x7f8868149dc0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 10 monclient: _renew_subs 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88681482c0 con 0x7f8868149dc0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f8870132640 1 librados: init done 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88640019d0 con 0x7f8868149dc0 2024-03-29T11:42:31.698 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f8864004aa0 con 0x7f8868149dc0 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885e7fc640 10 monclient: dump: 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:31.699 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.693+0000 7f885ffff640 4 set_mon_vals no callback set 2024-03-29T11:42:31.700 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.696+0000 7f8870132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f8864004c00 con 0x7f8868149dc0 2024-03-29T11:42:31.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.697+0000 7f8870132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f8870132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:31.701 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850005080 con 0x7f8868149dc0 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7f8864084b30 con 0x7f8868149dc0 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 10 client.4673.objecter ms_dispatch 0x7f8868000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.698+0000 7f885e7fc640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.699+0000 7f886dece640 10 monclient: get_auth_request con 0x7f884c06dd50 auth_method 0 2024-03-29T11:42:31.702 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.699+0000 7f885e7fc640 10 client.4673.objecter ms_handle_connect 0x7f884c06dd50 2024-03-29T11:42:31.709 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.706+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8864050410 con 0x7f8868149dc0 2024-03-29T11:42:31.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.706+0000 7f885e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.710 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:31.706+0000 7f885e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:32.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.040+0000 7f8870132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:32.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.040+0000 7f8870132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:32.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.040+0000 7f8870132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:32.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.040+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8850005330 con 0x7f8868149dc0 2024-03-29T11:42:32.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.041+0000 7f885e7fc640 1 -- 172.21.15.45:0/3624345515 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7f8864052340 con 0x7f8868149dc0 2024-03-29T11:42:32.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.041+0000 7f885e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:32.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.041+0000 7f885e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:32.045 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:32.045 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:32.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.046+0000 7f8870132640 10 librados: watch_flush enter 2024-03-29T11:42:32.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.046+0000 7f8870132640 10 librados: watch_flush exit 2024-03-29T11:42:32.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.046+0000 7f8870132640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:42:32.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.046+0000 7f8870132640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:42:32.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.047+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 >> v1:172.21.15.190:6816/1411598462 conn(0x7f884c06dd50 legacy=0x7f884c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.047+0000 7f8870132640 10 monclient: shutdown 2024-03-29T11:42:32.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.047+0000 7f8870132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:32.050 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.047+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 >> v1:172.21.15.45:6790/0 conn(0x7f8868149dc0 legacy=0x7f88681dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:32.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.048+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 shutdown_connections 2024-03-29T11:42:32.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.048+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 >> 172.21.15.45:0/3624345515 conn(0x7f88680b35e0 msgr2=0x7f886814a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:32.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.048+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 shutdown_connections 2024-03-29T11:42:32.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.048+0000 7f8870132640 1 -- 172.21.15.45:0/3624345515 wait complete. 2024-03-29T11:42:32.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:32.048+0000 7f8870132640 1 librados: shutdown 2024-03-29T11:42:32.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:38.877 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.029+0000 7f74d8182640 10 monclient: get_monmap_and_config 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.029+0000 7f74d8182640 10 monclient: build_initial_monmap 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.029+0000 7f74d8182640 10 monclient: monmap: 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:39.030834+0000 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:39.030834+0000 2024-03-29T11:42:39.033 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:39.034 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.034 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:39.034 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:39.034 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:39.034 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.035 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.030+0000 7f74d8182640 1 Processor -- start 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 1 -- start start 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient: init 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-a con 0x7f74d01457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-b con 0x7f74d013f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-c con 0x7f74d00a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d00aaef0 con 0x7f74d01457f0 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.031+0000 7f74d8182640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0145bd0 con 0x7f74d00a9d70 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d8182640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0145eb0 con 0x7f74d013f650 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d8182640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d8182640 10 monclient(hunting): authenticate will time out at 1415.271132s 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d671f640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f74d00a9d70 0x7f74d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.45:49652/0 (socket says 172.21.15.45:49652) 2024-03-29T11:42:39.036 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d671f640 1 -- 172.21.15.45:0/770658073 learned_addr learned my addr 172.21.15.45:0/770658073 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.032+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513030760 0 0) 0x7f74d0145bd0 con 0x7f74d00a9d70 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc003520 con 0x7f74d00a9d70 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701915207 0 0) 0x7f74d0145eb0 con 0x7f74d013f650 2024-03-29T11:42:39.037 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0145bd0 con 0x7f74d013f650 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386993590 0 0) 0x7f74d00aaef0 con 0x7f74d01457f0 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0145eb0 con 0x7f74d01457f0 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126150170 0 0) 0x7f74bc003520 con 0x7f74d00a9d70 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 >> v1:172.21.15.45:6790/0 conn(0x7f74d013f650 legacy=0x7f74d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 >> v1:172.21.15.45:6789/0 conn(0x7f74d01457f0 legacy=0x7f74d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: _finish_hunting 0 2024-03-29T11:42:39.038 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 monclient: found mon.noname-c 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0146340 con 0x7f74d00a9d70 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: _finish_auth 0 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: _check_auth_tickets 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74c8001940 con 0x7f74d00a9d70 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.033+0000 7f74d4f1c640 10 monclient: dump: 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.039 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.040 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.040 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.040 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.040 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.040 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:42:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74c8001b60 con 0x7f74d00a9d70 2024-03-29T11:42:39.043 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 4 set_mon_vals no callback set 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 1 -- 172.21.15.45:0/770658073 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74c8001ef0 con 0x7f74d00a9d70 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d4f1c640 10 monclient: dump: 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.044 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 10 monclient: shutdown 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.034+0000 7f74d8182640 1 -- 172.21.15.45:0/770658073 >> v1:172.21.15.190:6789/0 conn(0x7f74d00a9d70 legacy=0x7f74d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.035+0000 7f74d8182640 1 -- 172.21.15.45:0/770658073 shutdown_connections 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.035+0000 7f74d8182640 1 -- 172.21.15.45:0/770658073 >> 172.21.15.45:0/770658073 conn(0x7f74d0066a00 msgr2=0x7f74d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.035+0000 7f74d8182640 1 -- 172.21.15.45:0/770658073 shutdown_connections 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.035+0000 7f74d8182640 1 -- 172.21.15.45:0/770658073 wait complete. 2024-03-29T11:42:39.045 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.036+0000 7f74d8182640 10 monclient: build_initial_monmap 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.036+0000 7f74d8182640 10 monclient: monmap: 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 librados: starting msgr at 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 librados: starting objecter 2024-03-29T11:42:39.046 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 Processor -- start 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 -- start start 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 librados: setting wanted keys 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 1 librados: calling monclient init 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.037+0000 7f74d8182640 10 monclient: init 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d8182640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d8182640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-c con 0x7f74d00a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-a con 0x7f74d013f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d5f1e640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f74d00a9d70 0x7f74d01debc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:53720/0 (socket says 172.21.15.45:53720) 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.038+0000 7f74d5f1e640 1 -- 172.21.15.45:0/2137423150 learned_addr learned my addr 172.21.15.45:0/2137423150 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74c67fc640 10 client.?.objecter ms_handle_connect 0x7f74d00a9d70 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 10 monclient(hunting): picked mon.noname-b con 0x7f74d01457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0070a10 con 0x7f74d013f650 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d006f6f0 con 0x7f74d01457f0 2024-03-29T11:42:39.047 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74c67fc640 10 client.?.objecter ms_handle_connect 0x7f74d013f650 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d006f880 con 0x7f74d00a9d70 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.039+0000 7f74d8182640 10 monclient(hunting): authenticate will time out at 1415.278936s 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 client.?.objecter ms_handle_connect 0x7f74d01457f0 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699737567 0 0) 0x7f74d006f880 con 0x7f74d00a9d70 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b4004eb0 con 0x7f74d00a9d70 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586042344 0 0) 0x7f74d0070a10 con 0x7f74d013f650 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.048 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d006f880 con 0x7f74d013f650 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145621833 0 0) 0x7f74d006f6f0 con 0x7f74d01457f0 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:42:39.049 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0070a10 con 0x7f74d01457f0 2024-03-29T11:42:39.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.040+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283251954 0 0) 0x7f74b4004eb0 con 0x7f74d00a9d70 2024-03-29T11:42:39.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b4006680 con 0x7f74d00a9d70 2024-03-29T11:42:39.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 538899283 0 0) 0x7f74d006f880 con 0x7f74d013f650 2024-03-29T11:42:39.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b4004eb0 con 0x7f74d013f650 2024-03-29T11:42:39.051 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74c00018e0 con 0x7f74d00a9d70 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:39.052 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74cc001830 con 0x7f74d013f650 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411619414 0 0) 0x7f74d0070a10 con 0x7f74d01457f0 2024-03-29T11:42:39.053 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74d006f880 con 0x7f74d01457f0 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74c8001880 con 0x7f74d01457f0 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.054 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650019572 0 0) 0x7f74b4006680 con 0x7f74d00a9d70 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 >> v1:172.21.15.190:6789/0 conn(0x7f74d01457f0 legacy=0x7f74d01e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 >> v1:172.21.15.45:6789/0 conn(0x7f74d013f650 legacy=0x7f74d01df2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 monclient: found mon.c 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d01420a0 con 0x7f74d00a9d70 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74d8182640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.041+0000 7f74d8182640 10 monclient: _renew_subs 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:39.055 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d01422f0 con 0x7f74d00a9d70 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74c0001a40 con 0x7f74d00a9d70 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 10 monclient: _renew_subs 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d0070400 con 0x7f74d00a9d70 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f74c0003830 con 0x7f74d00a9d70 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74d8182640 1 librados: init done 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c7fff640 4 set_mon_vals no callback set 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.042+0000 7f74c67fc640 10 monclient: dump: 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:39.056 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.043+0000 7f74d8182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.043+0000 7f74d8182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.043+0000 7f74d8182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.043+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d0070ba0 con 0x7f74d00a9d70 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.044+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f74c0003a10 con 0x7f74d00a9d70 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7f74c0084a80 con 0x7f74d00a9d70 2024-03-29T11:42:39.057 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 10 client.4679.objecter ms_dispatch 0x7f74d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74d571d640 10 monclient: get_auth_request con 0x7f74b406e460 auth_method 0 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.045+0000 7f74c67fc640 10 client.4679.objecter ms_handle_connect 0x7f74b406e460 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.054+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74c0050190 con 0x7f74d00a9d70 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.054+0000 7f74c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:39.058 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.054+0000 7f74c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:39.429 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.426+0000 7f74d8182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.426+0000 7f74d8182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.426+0000 7f74d8182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:39.430 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.426+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74d0075f10 con 0x7f74d00a9d70 2024-03-29T11:42:39.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.428+0000 7f74c67fc640 1 -- 172.21.15.45:0/2137423150 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7f74c0052290 con 0x7f74d00a9d70 2024-03-29T11:42:39.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.428+0000 7f74c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.432 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.428+0000 7f74c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:39.432 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:39.432 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 10 librados: watch_flush enter 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 10 librados: watch_flush exit 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 10 client.4679.objecter successfully canceled tick 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 >> v1:172.21.15.190:6816/1411598462 conn(0x7f74b406e460 legacy=0x7f74b4070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.435 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 10 monclient: shutdown 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 >> v1:172.21.15.45:6790/0 conn(0x7f74d00a9d70 legacy=0x7f74d01debc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 shutdown_connections 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.432+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 >> 172.21.15.45:0/2137423150 conn(0x7f74d0066a00 msgr2=0x7f74d013ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.433+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 shutdown_connections 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.433+0000 7f74d8182640 1 -- 172.21.15.45:0/2137423150 wait complete. 2024-03-29T11:42:39.436 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:39.433+0000 7f74d8182640 1 librados: shutdown 2024-03-29T11:42:39.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:46.256 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:46.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.421+0000 7f388c9db640 10 monclient: get_monmap_and_config 2024-03-29T11:42:46.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.421+0000 7f388c9db640 10 monclient: build_initial_monmap 2024-03-29T11:42:46.424 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.421+0000 7f388c9db640 10 monclient: monmap: 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:46.422076+0000 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:46.422076+0000 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:46.425 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.423+0000 7f388c9db640 1 Processor -- start 2024-03-29T11:42:46.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.423+0000 7f388c9db640 1 -- start start 2024-03-29T11:42:46.426 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.423+0000 7f388c9db640 10 monclient: init 2024-03-29T11:42:46.437 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-c con 0x7f38881457f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-a con 0x7f388813f650 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-b con 0x7f38880a9d70 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880aaef0 con 0x7f388813f650 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888145bd0 con 0x7f38881457f0 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888145eb0 con 0x7f38880a9d70 2024-03-29T11:42:46.438 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.424+0000 7f388c9db640 10 monclient(hunting): authenticate will time out at 1422.663073s 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.425+0000 7f3886d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f38880a9d70 0x7f388814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:46936/0 (socket says 172.21.15.45:46936) 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.425+0000 7f3886d9d640 1 -- 172.21.15.45:0/4114746527 learned_addr learned my addr 172.21.15.45:0/4114746527 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638775520 0 0) 0x7f3888145eb0 con 0x7f38880a9d70 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:42:46.439 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.427+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f386c003520 con 0x7f38880a9d70 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458401016 0 0) 0x7f38880aaef0 con 0x7f388813f650 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.428+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888145eb0 con 0x7f388813f650 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608486181 0 0) 0x7f3888145eb0 con 0x7f388813f650 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 >> v1:172.21.15.45:6790/0 conn(0x7f38880a9d70 legacy=0x7f388814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 >> v1:172.21.15.190:6789/0 conn(0x7f38881457f0 legacy=0x7f388813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:46.440 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 monclient: found mon.noname-a 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888146340 con 0x7f388813f650 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: _finish_auth 0 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:46.441 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3878001670 con 0x7f388813f650 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 10 monclient: dump: 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:46.442 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.429+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3878001890 con 0x7f388813f650 2024-03-29T11:42:46.443 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388559a640 1 -- 172.21.15.45:0/4114746527 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3878001c20 con 0x7f388813f650 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388559a640 10 monclient: dump: 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.444 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388c9db640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388c9db640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.430+0000 7f388c9db640 4 set_mon_vals no callback set 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 10 monclient: shutdown 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 1 -- 172.21.15.45:0/4114746527 >> v1:172.21.15.45:6789/0 conn(0x7f388813f650 legacy=0x7f38880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 1 -- 172.21.15.45:0/4114746527 shutdown_connections 2024-03-29T11:42:46.445 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 1 -- 172.21.15.45:0/4114746527 >> 172.21.15.45:0/4114746527 conn(0x7f3888066a00 msgr2=0x7f3888066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 1 -- 172.21.15.45:0/4114746527 shutdown_connections 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.431+0000 7f388c9db640 1 -- 172.21.15.45:0/4114746527 wait complete. 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.433+0000 7f388c9db640 10 monclient: build_initial_monmap 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.433+0000 7f388c9db640 10 monclient: monmap: 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.446 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.433+0000 7f388c9db640 1 librados: starting msgr at 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.433+0000 7f388c9db640 1 librados: starting objecter 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 1 Processor -- start 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 1 -- start start 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 1 librados: setting wanted keys 2024-03-29T11:42:46.447 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 1 librados: calling monclient init 2024-03-29T11:42:46.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 10 monclient: init 2024-03-29T11:42:46.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:46.448 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.434+0000 7f388c9db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:46.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-b con 0x7f38880a9d70 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:46.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-c con 0x7f388813f650 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:46.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 10 monclient(hunting): picked mon.noname-a con 0x7f38881457f0 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:46.460 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888140250 con 0x7f38881457f0 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38881407d0 con 0x7f38880a9d70 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388813fda0 con 0x7f388813f650 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.435+0000 7f388c9db640 10 monclient(hunting): authenticate will time out at 1422.673903s 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.436+0000 7f3885d9b640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7f388813f650 0x7f3888143060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:47286/0 (socket says 172.21.15.45:47286) 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.436+0000 7f3885d9b640 1 -- 172.21.15.45:0/2827875135 learned_addr learned my addr 172.21.15.45:0/2827875135 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:46.461 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.436+0000 7f387effd640 10 client.?.objecter ms_handle_connect 0x7f388813f650 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.436+0000 7f387effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628884670 0 0) 0x7f388813fda0 con 0x7f388813f650 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:42:46.462 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.438+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3870004eb0 con 0x7f388813f650 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.439+0000 7f387effd640 10 client.?.objecter ms_handle_connect 0x7f38880a9d70 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.439+0000 7f387effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.439+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279495920 0 0) 0x7f3870004eb0 con 0x7f388813f650 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.439+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f388813fda0 con 0x7f388813f650 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.440+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3878001ce0 con 0x7f388813f650 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.440+0000 7f387effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.441+0000 7f387effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.442+0000 7f387effd640 10 monclient(hunting): dump: 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:46.463 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.442+0000 7f387effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.442+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002047745 0 0) 0x7f38881407d0 con 0x7f38880a9d70 2024-03-29T11:42:46.464 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3870004eb0 con 0x7f38880a9d70 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1638522110 0 0) 0x7f388813fda0 con 0x7f388813f650 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 >> v1:172.21.15.190:6789/0 conn(0x7f38880a9d70 legacy=0x7f3888134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 >> v1:172.21.15.45:6789/0 conn(0x7f38881457f0 legacy=0x7f3888143770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 monclient: found mon.c 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:46.465 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388813ff30 con 0x7f388813f650 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient: _finish_auth 0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.443+0000 7f387effd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f387effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 10 monclient: _renew_subs 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:46.466 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3888144f70 con 0x7f388813f650 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 10 monclient: _renew_subs 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3888135830 con 0x7f388813f650 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.444+0000 7f388c9db640 1 librados: init done 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38780035f0 con 0x7f388813f650 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7f3878003ca0 con 0x7f388813f650 2024-03-29T11:42:46.467 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.445+0000 7f387effd640 10 monclient: dump: 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.468 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.446+0000 7f387ffff640 4 set_mon_vals no callback set 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.446+0000 7f387cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.447+0000 7f387cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.447+0000 7f387cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.447+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3888075f10 con 0x7f388813f650 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.448+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7f3878003e00 con 0x7f388813f650 2024-03-29T11:42:46.469 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.448+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7f3878084c30 con 0x7f388813f650 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.448+0000 7f387effd640 10 client.4685.objecter ms_dispatch 0x7f3888000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.448+0000 7f387effd640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.448+0000 7f387effd640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.449+0000 7f387effd640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T11:42:46.470 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.449+0000 7f388659c640 10 monclient: get_auth_request con 0x7f387006dde0 auth_method 0 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.450+0000 7f387effd640 10 client.4685.objecter ms_handle_connect 0x7f387006dde0 2024-03-29T11:42:46.471 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.458+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3878050510 con 0x7f388813f650 2024-03-29T11:42:46.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.458+0000 7f387effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.472 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.458+0000 7f387effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:46.799 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.797+0000 7f387cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.797+0000 7f387cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.797+0000 7f387cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:46.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.797+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38881400c0 con 0x7f388813f650 2024-03-29T11:42:46.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.804+0000 7f387effd640 1 -- 172.21.15.45:0/2827875135 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7f3878052440 con 0x7f388813f650 2024-03-29T11:42:46.807 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:46.807 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:46.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.804+0000 7f387effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.804+0000 7f387effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:46.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.808+0000 7f387cff9640 10 librados: watch_flush enter 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.810+0000 7f387cff9640 10 librados: watch_flush exit 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.810+0000 7f387cff9640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.810+0000 7f387cff9640 10 client.4685.objecter successfully canceled tick 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.811+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 >> v1:172.21.15.190:6816/1411598462 conn(0x7f387006dde0 legacy=0x7f38700702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.811+0000 7f387cff9640 10 monclient: shutdown 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.811+0000 7f387cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.811+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 >> v1:172.21.15.45:6790/0 conn(0x7f388813f650 legacy=0x7f3888143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.813+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 shutdown_connections 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.813+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 >> 172.21.15.45:0/2827875135 conn(0x7f3888066a00 msgr2=0x7f388813b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.813+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 shutdown_connections 2024-03-29T11:42:46.816 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.813+0000 7f387cff9640 1 -- 172.21.15.45:0/2827875135 wait complete. 2024-03-29T11:42:46.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:46.813+0000 7f387cff9640 1 librados: shutdown 2024-03-29T11:42:46.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:53.643 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:53.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.790+0000 7ff56904d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:53.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.790+0000 7ff56904d640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.793 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.790+0000 7ff56904d640 10 monclient: monmap: 2024-03-29T11:42:53.793 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:42:53.791242+0000 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:42:53.791242+0000 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:42:53.794 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.791+0000 7ff56904d640 1 Processor -- start 2024-03-29T11:42:53.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.792+0000 7ff56904d640 1 -- start start 2024-03-29T11:42:53.795 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.792+0000 7ff56904d640 10 monclient: init 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.792+0000 7ff56904d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5640b4790 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5640aea90 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640b39f0 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640b0c60 con 0x7ff5640aea90 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640af020 con 0x7ff5640b39f0 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56413fbd0 con 0x7ff5640b4790 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.797 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff56904d640 10 monclient(hunting): authenticate will time out at 1430.032197s 2024-03-29T11:42:53.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff562d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff5640b4790 0x7ff5640b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:55926/0 (socket says 172.21.15.45:55926) 2024-03-29T11:42:53.798 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.793+0000 7ff562d9d640 1 -- 172.21.15.45:0/228145761 learned_addr learned my addr 172.21.15.45:0/228145761 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:53.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335779317 0 0) 0x7ff56413fbd0 con 0x7ff5640b4790 2024-03-29T11:42:53.801 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:42:53.802 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff54c003520 con 0x7ff5640b4790 2024-03-29T11:42:53.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 665834725 0 0) 0x7ff54c003520 con 0x7ff5640b4790 2024-03-29T11:42:53.803 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 >> v1:172.21.15.190:6789/0 conn(0x7ff5640b39f0 legacy=0x7ff56413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 >> v1:172.21.15.45:6789/0 conn(0x7ff5640aea90 legacy=0x7ff5640af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 monclient: found mon.noname-b 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56413fd60 con 0x7ff5640b4790 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff554001690 con 0x7ff5640b4790 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.798+0000 7ff561d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: dump: 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:53.804 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff56904d640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff56904d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff554001920 con 0x7ff5640b4790 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 1 -- 172.21.15.45:0/228145761 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff554001cf0 con 0x7ff5640b4790 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.805 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff561d9b640 10 monclient: dump: 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff56904d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.799+0000 7ff56904d640 4 set_mon_vals no callback set 2024-03-29T11:42:53.806 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.800+0000 7ff56904d640 10 monclient: shutdown 2024-03-29T11:42:53.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.800+0000 7ff56904d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.800+0000 7ff56904d640 1 -- 172.21.15.45:0/228145761 >> v1:172.21.15.45:6790/0 conn(0x7ff5640b4790 legacy=0x7ff5640b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.800+0000 7ff56904d640 1 -- 172.21.15.45:0/228145761 shutdown_connections 2024-03-29T11:42:53.807 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.800+0000 7ff56904d640 1 -- 172.21.15.45:0/228145761 >> 172.21.15.45:0/228145761 conn(0x7ff564066a00 msgr2=0x7ff564066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.801+0000 7ff56904d640 1 -- 172.21.15.45:0/228145761 shutdown_connections 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.801+0000 7ff56904d640 1 -- 172.21.15.45:0/228145761 wait complete. 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.802+0000 7ff56904d640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.802+0000 7ff56904d640 10 monclient: monmap: 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:42:53.809 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.802+0000 7ff56904d640 1 librados: starting msgr at 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.802+0000 7ff56904d640 1 librados: starting objecter 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 Processor -- start 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 -- start start 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 librados: setting wanted keys 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 librados: calling monclient init 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient: init 2024-03-29T11:42:53.810 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640b39f0 addr v1:172.21.15.45:6790/0 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5640b4790 addr v1:172.21.15.190:6789/0 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5640aad30 addr v1:172.21.15.45:6789/0 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640ae530 con 0x7ff5640aad30 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640ad000 con 0x7ff5640b4790 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564131930 con 0x7ff5640b39f0 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.811 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.803+0000 7ff56904d640 10 monclient(hunting): authenticate will time out at 1430.042988s 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.806+0000 7ff562d9d640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff5640b39f0 0x7ff5641e7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:56114/0 (socket says 172.21.15.45:56114) 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.806+0000 7ff562d9d640 1 -- 172.21.15.45:0/3194585177 learned_addr learned my addr 172.21.15.45:0/3194585177 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 client.?.objecter ms_handle_connect 0x7ff5640b39f0 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582690490 0 0) 0x7ff564131930 con 0x7ff5640b39f0 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550004eb0 con 0x7ff5640b39f0 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 client.?.objecter ms_handle_connect 0x7ff5640b4790 2024-03-29T11:42:53.817 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.807+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678222391 0 0) 0x7ff550004eb0 con 0x7ff5640b39f0 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff564131930 con 0x7ff5640b39f0 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff554001cf0 con 0x7ff5640b39f0 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:53.818 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.808+0000 7ff54b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.809+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1243309195 0 0) 0x7ff564131930 con 0x7ff5640b39f0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.809+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 >> v1:172.21.15.190:6789/0 conn(0x7ff5640b4790 legacy=0x7ff5641e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.809+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 >> v1:172.21.15.45:6789/0 conn(0x7ff5640aad30 legacy=0x7ff5641ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 1 monclient: found mon.c 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564135700 con 0x7ff5640b39f0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.819 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 10 monclient: _renew_subs 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5640ad410 con 0x7ff5640b39f0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 10 monclient: _renew_subs 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5641308f0 con 0x7ff5640b39f0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.810+0000 7ff56904d640 1 librados: init done 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff554004a90 con 0x7ff5640b39f0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff554004e60 con 0x7ff5640b39f0 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff560d99640 4 set_mon_vals no callback set 2024-03-29T11:42:53.820 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.811+0000 7ff54b7fe640 10 monclient: dump: 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.812+0000 7ff56904d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.812+0000 7ff56904d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.821 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.812+0000 7ff56904d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.812+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544005080 con 0x7ff5640b39f0 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7ff554004fc0 con 0x7ff5640b39f0 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7ff554084c60 con 0x7ff5640b39f0 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 10 client.4691.objecter ms_dispatch 0x7ff564000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff54b7fe640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-29T11:42:53.822 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.813+0000 7ff56259c640 10 monclient: get_auth_request con 0x7ff55006dbd0 auth_method 0 2024-03-29T11:42:53.823 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.814+0000 7ff54b7fe640 10 client.4691.objecter ms_handle_connect 0x7ff55006dbd0 2024-03-29T11:42:53.824 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.821+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff554050540 con 0x7ff5640b39f0 2024-03-29T11:42:53.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.821+0000 7ff54b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.825 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:53.821+0000 7ff54b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:54.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.201+0000 7ff56904d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:54.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.201+0000 7ff56904d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:54.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.201+0000 7ff56904d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:42:54.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.201+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff544005330 con 0x7ff5640b39f0 2024-03-29T11:42:54.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.203+0000 7ff54b7fe640 1 -- 172.21.15.45:0/3194585177 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7ff554052470 con 0x7ff5640b39f0 2024-03-29T11:42:54.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.204+0000 7ff54b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:54.207 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.204+0000 7ff54b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:54.207 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:42:54.207 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:54.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.207+0000 7ff56904d640 10 librados: watch_flush enter 2024-03-29T11:42:54.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.207+0000 7ff56904d640 10 librados: watch_flush exit 2024-03-29T11:42:54.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.207+0000 7ff56904d640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-29T11:42:54.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.207+0000 7ff56904d640 10 client.4691.objecter successfully canceled tick 2024-03-29T11:42:54.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.208+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 >> v1:172.21.15.190:6816/1411598462 conn(0x7ff55006dbd0 legacy=0x7ff550070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 10 monclient: shutdown 2024-03-29T11:42:54.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:54.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 >> v1:172.21.15.45:6790/0 conn(0x7ff5640b39f0 legacy=0x7ff5641e7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:54.213 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 shutdown_connections 2024-03-29T11:42:54.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 >> 172.21.15.45:0/3194585177 conn(0x7ff564066a00 msgr2=0x7ff5640b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:54.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 shutdown_connections 2024-03-29T11:42:54.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.209+0000 7ff56904d640 1 -- 172.21.15.45:0/3194585177 wait complete. 2024-03-29T11:42:54.214 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:42:54.210+0000 7ff56904d640 1 librados: shutdown 2024-03-29T11:42:54.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:01.041 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:01.179 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.176+0000 7ff219ba8640 10 monclient: get_monmap_and_config 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.176+0000 7ff219ba8640 10 monclient: build_initial_monmap 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.176+0000 7ff219ba8640 10 monclient: monmap: 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:43:01.177595+0000 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:43:01.177595+0000 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:43:01.180 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.181 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.178+0000 7ff219ba8640 1 Processor -- start 2024-03-29T11:43:01.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.178+0000 7ff219ba8640 1 -- start start 2024-03-29T11:43:01.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.178+0000 7ff219ba8640 10 monclient: init 2024-03-29T11:43:01.182 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.179+0000 7ff219ba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.179+0000 7ff219ba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.179+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-a con 0x7ff2140b1f60 addr v1:172.21.15.45:6789/0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.179+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-c con 0x7ff214071eb0 addr v1:172.21.15.190:6789/0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-b con 0x7ff214143d10 addr v1:172.21.15.45:6790/0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21406ef80 con 0x7ff2140b1f60 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21406f260 con 0x7ff214071eb0 2024-03-29T11:43:01.183 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140b40c0 con 0x7ff214143d10 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff219ba8640 10 monclient(hunting): authenticate will time out at 1437.419553s 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff213fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff214143d10 0x7ff214146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:38984/0 (socket says 172.21.15.45:38984) 2024-03-29T11:43:01.184 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.180+0000 7ff213fff640 1 -- 172.21.15.45:0/1037925409 learned_addr learned my addr 172.21.15.45:0/1037925409 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884980878 0 0) 0x7ff2140b40c0 con 0x7ff214143d10 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:43:01.185 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff200003520 con 0x7ff214143d10 2024-03-29T11:43:01.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352446372 0 0) 0x7ff200003520 con 0x7ff214143d10 2024-03-29T11:43:01.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 >> v1:172.21.15.190:6789/0 conn(0x7ff214071eb0 legacy=0x7ff214146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:01.186 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.182+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 >> v1:172.21.15.45:6789/0 conn(0x7ff2140b1f60 legacy=0x7ff214143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214155d70 con 0x7ff214143d10 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:01.188 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff20c001a10 con 0x7ff214143d10 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: dump: 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:01.189 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 <== mon.2 v1:172.21.15.45:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff20c001be0 con 0x7ff214143d10 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 1 -- 172.21.15.45:0/1037925409 <== mon.2 v1:172.21.15.45:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff20c002030 con 0x7ff214143d10 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff212ffd640 10 monclient: dump: 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:01.190 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff219ba8640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff219ba8640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.183+0000 7ff219ba8640 4 set_mon_vals no callback set 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.184+0000 7ff219ba8640 10 monclient: shutdown 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.184+0000 7ff219ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.184+0000 7ff219ba8640 1 -- 172.21.15.45:0/1037925409 >> v1:172.21.15.45:6790/0 conn(0x7ff214143d10 legacy=0x7ff214146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.185+0000 7ff219ba8640 1 -- 172.21.15.45:0/1037925409 shutdown_connections 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.185+0000 7ff219ba8640 1 -- 172.21.15.45:0/1037925409 >> 172.21.15.45:0/1037925409 conn(0x7ff214066a00 msgr2=0x7ff214066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:01.191 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.187+0000 7ff219ba8640 1 -- 172.21.15.45:0/1037925409 shutdown_connections 2024-03-29T11:43:01.192 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.187+0000 7ff219ba8640 1 -- 172.21.15.45:0/1037925409 wait complete. 2024-03-29T11:43:01.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.188+0000 7ff219ba8640 10 monclient: build_initial_monmap 2024-03-29T11:43:01.193 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 10 monclient: monmap: 2024-03-29T11:43:01.193 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 librados: starting msgr at 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 librados: starting objecter 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 Processor -- start 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 -- start start 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 librados: setting wanted keys 2024-03-29T11:43:01.194 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 1 librados: calling monclient init 2024-03-29T11:43:01.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.189+0000 7ff219ba8640 10 monclient: init 2024-03-29T11:43:01.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.190+0000 7ff219ba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:01.195 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.190+0000 7ff219ba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.190+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-a con 0x7ff2140b1f60 addr v1:172.21.15.45:6789/0 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-b con 0x7ff214143d10 addr v1:172.21.15.190:6789/0 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 10 monclient(hunting): picked mon.noname-c con 0x7ff214071180 addr v1:172.21.15.45:6790/0 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21406f800 con 0x7ff2140b1f60 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff214140b70 con 0x7ff214143d10 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2141dfa70 con 0x7ff214071180 2024-03-29T11:43:01.196 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff219ba8640 10 monclient(hunting): authenticate will time out at 1437.430494s 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff213fff640 1 --1- >> v1:172.21.15.45:6790/0 conn(0x7ff214071180 0x7ff2141df360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6790/0 says I am v1:172.21.15.45:39250/0 (socket says 172.21.15.45:39250) 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff213fff640 1 -- 172.21.15.45:0/1702137170 learned_addr learned my addr 172.21.15.45:0/1702137170 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff210ff9640 10 client.?.objecter ms_handle_connect 0x7ff214071180 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.191+0000 7ff210ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135814628 0 0) 0x7ff2141dfa70 con 0x7ff214071180 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:43:01.197 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.192+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f4004eb0 con 0x7ff214071180 2024-03-29T11:43:01.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 client.?.objecter ms_handle_connect 0x7ff2140b1f60 2024-03-29T11:43:01.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:01.199 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118455593 0 0) 0x7ff1f4004eb0 con 0x7ff214071180 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2141dfa70 con 0x7ff214071180 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff20c001f30 con 0x7ff214071180 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:01.200 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.194+0000 7ff210ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073268146 0 0) 0x7ff21406f800 con 0x7ff2140b1f60 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f4004eb0 con 0x7ff2140b1f60 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3002897014 0 0) 0x7ff2141dfa70 con 0x7ff214071180 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 >> v1:172.21.15.190:6789/0 conn(0x7ff214143d10 legacy=0x7ff2141dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 >> v1:172.21.15.45:6789/0 conn(0x7ff2140b1f60 legacy=0x7ff214154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 monclient: found mon.c 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:01.201 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214070150 con 0x7ff214071180 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.195+0000 7ff210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 5 monclient: authenticate success, global_id 4697 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 10 monclient: _renew_subs 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff214070430 con 0x7ff214071180 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 10 monclient: _renew_subs 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff214154350 con 0x7ff214071180 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff219ba8640 1 librados: init done 2024-03-29T11:43:01.202 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff20c004e80 con 0x7ff214071180 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7ff20c0050f0 con 0x7ff214071180 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff210ff9640 10 monclient: dump: 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:01.203 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.196+0000 7ff2127fc640 4 set_mon_vals no callback set 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7ff20c005250 con 0x7ff214071180 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7ff20c084e40 con 0x7ff214071180 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 10 client.4697.objecter ms_dispatch 0x7ff214000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:43:01.204 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff210ff9640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.198+0000 7ff2137fe640 10 monclient: get_auth_request con 0x7ff1f406de00 auth_method 0 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.199+0000 7ff210ff9640 10 client.4697.objecter ms_handle_connect 0x7ff1f406de00 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.199+0000 7ff219ba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.199+0000 7ff219ba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.199+0000 7ff219ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.205 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.199+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f8005080 con 0x7ff214071180 2024-03-29T11:43:01.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.207+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff20c050720 con 0x7ff214071180 2024-03-29T11:43:01.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.207+0000 7ff210ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:01.211 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.207+0000 7ff210ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:01.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.531+0000 7ff219ba8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.531+0000 7ff219ba8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.531+0000 7ff219ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:01.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.531+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1f8005330 con 0x7ff214071180 2024-03-29T11:43:01.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.532+0000 7ff210ff9640 1 -- 172.21.15.45:0/1702137170 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7ff20c052650 con 0x7ff214071180 2024-03-29T11:43:01.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.532+0000 7ff210ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.535 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.532+0000 7ff210ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:01.537 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:43:01.537 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.535+0000 7ff219ba8640 10 librados: watch_flush enter 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.535+0000 7ff219ba8640 10 librados: watch_flush exit 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.535+0000 7ff219ba8640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.535+0000 7ff219ba8640 10 client.4697.objecter successfully canceled tick 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.535+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 >> v1:172.21.15.190:6816/1411598462 conn(0x7ff1f406de00 legacy=0x7ff1f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.536+0000 7ff219ba8640 10 monclient: shutdown 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.536+0000 7ff219ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:01.539 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.536+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 >> v1:172.21.15.45:6790/0 conn(0x7ff214071180 legacy=0x7ff2141df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.538+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 shutdown_connections 2024-03-29T11:43:01.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.538+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 >> 172.21.15.45:0/1702137170 conn(0x7ff214066a00 msgr2=0x7ff2140b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:01.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.538+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 shutdown_connections 2024-03-29T11:43:01.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.538+0000 7ff219ba8640 1 -- 172.21.15.45:0/1702137170 wait complete. 2024-03-29T11:43:01.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:01.539+0000 7ff219ba8640 1 librados: shutdown 2024-03-29T11:43:01.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:08.375 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:08.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.530+0000 7fe9f69f6640 10 monclient: get_monmap_and_config 2024-03-29T11:43:08.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.530+0000 7fe9f69f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.534 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.530+0000 7fe9f69f6640 10 monclient: monmap: 2024-03-29T11:43:08.534 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:43:08.531840+0000 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:43:08.531840+0000 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-b 2024-03-29T11:43:08.535 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.531+0000 7fe9f69f6640 1 Processor -- start 2024-03-29T11:43:08.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 1 -- start start 2024-03-29T11:43:08.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient: init 2024-03-29T11:43:08.536 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:08.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f01422b0 addr v1:172.21.15.45:6789/0 2024-03-29T11:43:08.540 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f0142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f00b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f0142c80 con 0x7fe9f01422b0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00aae40 con 0x7fe9f00b17d0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f013fbd0 con 0x7fe9f0142690 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.532+0000 7fe9f69f6640 10 monclient(hunting): authenticate will time out at 1444.772057s 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9effff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe9f01422b0 0x7fe9f013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:55722/0 (socket says 172.21.15.45:55722) 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9effff640 1 -- 172.21.15.45:0/2653243183 learned_addr learned my addr 172.21.15.45:0/2653243183 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003565744 0 0) 0x7fe9f0142c80 con 0x7fe9f01422b0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.533+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc003520 con 0x7fe9f01422b0 2024-03-29T11:43:08.541 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 <== mon.0 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300361318 0 0) 0x7fe9dc003520 con 0x7fe9f01422b0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 >> v1:172.21.15.45:6790/0 conn(0x7fe9f0142690 legacy=0x7fe9f00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 >> v1:172.21.15.190:6789/0 conn(0x7fe9f00b17d0 legacy=0x7fe9f00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.45:6789/0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f0151940 con 0x7fe9f01422b0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9f69f6640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9f69f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 <== mon.0 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe9e0001690 con 0x7fe9f01422b0 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.542 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9ef7fe640 10 monclient: dump: 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.534+0000 7fe9f69f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 <== mon.0 v1:172.21.15.45:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e00018b0 con 0x7fe9f01422b0 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.543 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9f69f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9f69f6640 4 set_mon_vals no callback set 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 1 -- 172.21.15.45:0/2653243183 <== mon.0 v1:172.21.15.45:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe9e0001bc0 con 0x7fe9f01422b0 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.535+0000 7fe9ef7fe640 10 monclient: dump: 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.544 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.545 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.545 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 10 monclient: shutdown 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 1 -- 172.21.15.45:0/2653243183 >> v1:172.21.15.45:6789/0 conn(0x7fe9f01422b0 legacy=0x7fe9f013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 1 -- 172.21.15.45:0/2653243183 shutdown_connections 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 1 -- 172.21.15.45:0/2653243183 >> 172.21.15.45:0/2653243183 conn(0x7fe9f0066a00 msgr2=0x7fe9f0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 1 -- 172.21.15.45:0/2653243183 shutdown_connections 2024-03-29T11:43:08.546 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.536+0000 7fe9f69f6640 1 -- 172.21.15.45:0/2653243183 wait complete. 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 10 monclient: monmap: 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:epoch 0 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:last_changed 0.000000 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:created 0.000000 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.noname-a 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.noname-c 2024-03-29T11:43:08.552 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.553 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 librados: starting msgr at 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 librados: starting objecter 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 Processor -- start 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 -- start start 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 librados: setting wanted keys 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 1 librados: calling monclient init 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.537+0000 7fe9f69f6640 10 monclient: init 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.554 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f00b17d0 addr v1:172.21.15.190:6789/0 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f0142690 addr v1:172.21.15.45:6790/0 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f00ab540 addr v1:172.21.15.45:6789/0 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00abef0 con 0x7fe9f00ab540 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00acef0 con 0x7fe9f00b17d0 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 1 -- --> v1:172.21.15.45:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00ad1d0 con 0x7fe9f0142690 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f69f6640 10 monclient(hunting): authenticate will time out at 1444.777861s 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f4f93640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fe9f00ab540 0x7fe9f012fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:56146/0 (socket says 172.21.15.45:56146) 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.538+0000 7fe9f4f93640 1 -- 172.21.15.45:0/4001305891 learned_addr learned my addr 172.21.15.45:0/4001305891 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter ms_handle_connect 0x7fe9f00ab540 2024-03-29T11:43:08.555 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter ms_handle_connect 0x7fe9f0142690 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter ms_handle_connect 0x7fe9f00b17d0 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330823061 0 0) 0x7fe9f00ad1d0 con 0x7fe9f0142690 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.539+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d0004eb0 con 0x7fe9f0142690 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334106522 0 0) 0x7fe9f00acef0 con 0x7fe9f00b17d0 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.556 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d00051d0 con 0x7fe9f00b17d0 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508768661 0 0) 0x7fe9d0004eb0 con 0x7fe9f0142690 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9f00acef0 con 0x7fe9f0142690 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.0 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953939257 0 0) 0x7fe9f00abef0 con 0x7fe9f00ab540 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d0004eb0 con 0x7fe9f00ab540 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3706700192 0 0) 0x7fe9d00051d0 con 0x7fe9f00b17d0 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9f00abef0 con 0x7fe9f00b17d0 2024-03-29T11:43:08.557 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe9d8001810 con 0x7fe9f0142690 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:08.558 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe9e0001550 con 0x7fe9f00b17d0 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.540+0000 7fe9ed7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:08.559 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2516280797 0 0) 0x7fe9f00acef0 con 0x7fe9f0142690 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 >> v1:172.21.15.190:6789/0 conn(0x7fe9f00b17d0 legacy=0x7fe9f0132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 >> v1:172.21.15.45:6789/0 conn(0x7fe9f00ab540 legacy=0x7fe9f012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 1 monclient: found mon.c 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00ab9d0 con 0x7fe9f0142690 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.560 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 5 monclient: authenticate success, global_id 4703 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 10 monclient: _renew_subs 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f012f3c0 con 0x7fe9f0142690 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 10 monclient: _renew_subs 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f0130490 con 0x7fe9f0142690 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.541+0000 7fe9f69f6640 1 librados: init done 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.542+0000 7fe9f69f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.542+0000 7fe9f69f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.542+0000 7fe9f69f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:08.561 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.542+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f012f6c0 con 0x7fe9f0142690 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9d8001970 con 0x7fe9f0142690 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1873785690 0 0) 0x7fe9d80037b0 con 0x7fe9f0142690 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9ed7fa640 10 monclient: dump: 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:epoch 1 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:fsid 40d9e350-dd6b-4b6b-943c-8e6b711fa5c5 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:last_changed 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:created 2024-03-29T11:31:34.458558+0000 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:election_strategy: 1 2024-03-29T11:43:08.562 INFO:teuthology.misc.health.smithi045.stderr:0: v1:172.21.15.45:6789/0 mon.a 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2: v1:172.21.15.45:6790/0 mon.c 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr: 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.543+0000 7fe9eeffd640 4 set_mon_vals no callback set 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.545+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4284645070 0 0) 0x7fe9d8019780 con 0x7fe9f0142690 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4278036324 0 0) 0x7fe9d8083600 con 0x7fe9f0142690 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9ed7fa640 10 client.4703.objecter ms_dispatch 0x7fe9f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9ed7fa640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9ed7fa640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9ed7fa640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.546+0000 7fe9effff640 10 monclient: get_auth_request con 0x7fe9d006e1b0 auth_method 0 2024-03-29T11:43:08.563 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.547+0000 7fe9ed7fa640 10 client.4703.objecter ms_handle_connect 0x7fe9d006e1b0 2024-03-29T11:43:08.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.553+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9d804ef40 con 0x7fe9f0142690 2024-03-29T11:43:08.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.553+0000 7fe9ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.564 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.553+0000 7fe9ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:08.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.924+0000 7fe9f69f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.924+0000 7fe9f69f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.924+0000 7fe9f69f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.45:6790/0 2024-03-29T11:43:08.928 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.924+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 --> v1:172.21.15.45:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9f00abb60 con 0x7fe9f0142690 2024-03-29T11:43:08.930 INFO:teuthology.misc.health.smithi045.stdout: 2024-03-29T11:43:08.930 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":11,"fsid":"40d9e350-dd6b-4b6b-943c-8e6b711fa5c5","created":"2024-03-29T11:32:10.847603+0000","modified":"2024-03-29T11:42:15.774526+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:42:15.774526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1e4c9cf3-da32-4ce1-a5eb-5e192a28f1a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4dc754de-2886-4a2f-8cd4-5c2501bfd966","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d3c56240-80be-4874-b7a7-1895d9be2c40","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c06dd1-de39-4519-a58d-82501385cd63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"aa6ed173-15f2-407a-823d-f5da2f886d5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"86551d84-ea1f-4a90-b4d0-62b7dbff72b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7766d34a-8266-4da6-9cef-bf66289c4ce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b3d19353-b793-4b49-9361-aaf623b141ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:08.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.926+0000 7fe9ed7fa640 1 -- 172.21.15.45:0/4001305891 <== mon.2 v1:172.21.15.45:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 657713222) 0x7fe9d804ef40 con 0x7fe9f0142690 2024-03-29T11:43:08.930 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.926+0000 7fe9ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.931 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.926+0000 7fe9ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:08.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 10 librados: watch_flush enter 2024-03-29T11:43:08.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 10 librados: watch_flush exit 2024-03-29T11:43:08.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T11:43:08.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 10 client.4703.objecter successfully canceled tick 2024-03-29T11:43:08.935 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 >> v1:172.21.15.190:6816/1411598462 conn(0x7fe9d006e1b0 legacy=0x7fe9d0070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 10 monclient: shutdown 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 >> v1:172.21.15.45:6790/0 conn(0x7fe9f0142690 legacy=0x7fe9f012ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 shutdown_connections 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.930+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 >> 172.21.15.45:0/4001305891 conn(0x7fe9f0066a00 msgr2=0x7fe9f00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.931+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 shutdown_connections 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.931+0000 7fe9f69f6640 1 -- 172.21.15.45:0/4001305891 wait complete. 2024-03-29T11:43:08.936 INFO:teuthology.misc.health.smithi045.stderr:2024-03-29T11:43:08.931+0000 7fe9f69f6640 1 librados: shutdown 2024-03-29T11:43:08.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:08.950 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:43:08.952 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:43:08.953 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:43:08.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:08.982 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:43:08.982 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:43:08.983 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.062 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:43:09.062 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:43:09.062 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.135 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:43:09.135 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:43:09.135 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.215 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:43:09.217 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:43:09.217 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.252 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:43:09.252 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:43:09.252 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.325 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:43:09.325 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:43:09.325 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.395 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:43:09.395 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:43:09.395 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.471 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:43:09.471 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:43:09.472 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:43:09.472 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.496 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:43:09.496 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:43:09.496 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.539 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:43:09.539 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:43:09.539 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:43:09.539 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.550 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:43:09.551 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:43:09.551 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.602 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:43:09.603 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:43:09.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:43:09.615 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:43:09.615 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:43:09.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:43:09.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:43:09.677 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:43:10.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:43:10.668 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:43:10.781 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:43:10.781 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:43:10.956 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:43:10.957 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:43:11.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:43:11.074 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:43:14.575 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:43:14.575 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:43:14.638 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:43:14.638 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:43:14.702 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:43:14.702 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:43:14.762 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:43:14.763 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/data/mon.a.tgz 2024-03-29T11:43:14.764 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T11:43:14.781 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.ySNG2iYvh4 2024-03-29T11:43:14.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ySNG2iYvh4 2024-03-29T11:43:14.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.ySNG2iYvh4 2024-03-29T11:43:14.988 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.ySNG2iYvh4 is 211KB 2024-03-29T11:43:15.082 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.ySNG2iYvh4 2024-03-29T11:43:15.097 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/data/mon.c.tgz 2024-03-29T11:43:15.097 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-29T11:43:15.151 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.ZdjfYZeHgh 2024-03-29T11:43:15.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ZdjfYZeHgh 2024-03-29T11:43:15.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.ZdjfYZeHgh 2024-03-29T11:43:15.339 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.ZdjfYZeHgh is 211KB 2024-03-29T11:43:15.431 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.ZdjfYZeHgh 2024-03-29T11:43:15.445 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/data/mon.b.tgz 2024-03-29T11:43:15.445 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-29T11:43:15.463 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.GpUPMVZSR4 2024-03-29T11:43:15.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GpUPMVZSR4 2024-03-29T11:43:15.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.GpUPMVZSR4 2024-03-29T11:43:15.670 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.GpUPMVZSR4 is 211KB 2024-03-29T11:43:15.765 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.GpUPMVZSR4 2024-03-29T11:43:15.780 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:43:15.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:43:15.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:43:15.863 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:43:15.903 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:43:15.903 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:43:15.905 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi045/crash 2024-03-29T11:43:15.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:43:15.933 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi190/crash 2024-03-29T11:43:15.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:43:15.971 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:43:15.971 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:43:15.977 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:43:16.003 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-03-29T11:43:16.003 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:43:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-03-29T11:43:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-03-29T11:43:16.004 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-03-29T11:43:16.004 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39272.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39935.log 2024-03-29T11:43:16.042 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39545.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39194.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39935.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39194.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39545.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41500.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39194.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40052.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35216.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-29T11:43:16.043 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41773.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40052.log: /var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40091.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40052.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.41773.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38804.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40091.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41617.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38882.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38804.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-03-29T11:43:16.044 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38804.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.38882.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39506.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41539.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39506.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39389.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-29T11:43:16.045 INFO:teuthology.orchestra.run.smithi045.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39038.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39389.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39233.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39584.log 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2024-03-29T11:43:16.046 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41968.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38921.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39584.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38607.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38921.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38529.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38529.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41578.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38529.log.gz 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2024-03-29T11:43:16.047 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41030.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41578.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39155.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39350.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40481.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41695.log 2024-03-29T11:43:16.048 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40013.log: /var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41734.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40598.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41695.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41734.log: /var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38685.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41734.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41929.log 2024-03-29T11:43:16.049 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39623.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40130.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41929.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38568.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39623.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40130.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40130.log.gz 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38568.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38960.log 2024-03-29T11:43:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40442.log: gzip -5 --verbose 87.1% -- /var/log/ceph/ceph-client.admin.39116.log -- replaced with /var/log/ceph/ceph-client.admin.38568.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39818.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38960.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40286.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39116.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39311.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39662.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39818.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39311.log: gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40286.log.gz 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: /var/log/ceph/ceph-client.admin.38724.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39779.log 2024-03-29T11:43:16.051 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39311.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39662.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39779.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39974.log: /var/log/ceph/ceph-client.admin.38843.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39779.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41187.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35319.log 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 86.6% 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-03-29T11:43:16.052 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.35319.log: gzip -5 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41187.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39740.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35319.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41343.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39857.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40757.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39857.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39857.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr: -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40835.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:43:16.053 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.39701.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40325.log: /var/log/ceph/ceph.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41890.log: /var/log/ceph/ceph-client.admin.39701.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39701.log.gz 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38646.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29094.log 2024-03-29T11:43:16.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38646.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.tmp-client.admin.29094.log: 0.0% 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38646.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.29094.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40208.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.38763.log: /var/log/ceph/ceph-client.admin.38999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39467.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.40208.log: 85.8% 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40208.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.39467.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.41851.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39467.log.gz 2024-03-29T11:43:16.055 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2024-03-29T11:43:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:43:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-03-29T11:43:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:43:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34988.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:43:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35028.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35108.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35108.log: -- /var/log/ceph/ceph-client.admin.34948.log 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35068.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35108.log.gz 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:43:16.063 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34948.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35188.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34948.log.gz 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr:gzip 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.35228.log 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-03-29T11:43:16.064 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35228.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35228.log.gz 2024-03-29T11:43:16.116 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:43:27.910 INFO:teuthology.orchestra.run.smithi190.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:43:27.937 INFO:teuthology.orchestra.run.smithi190.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:43:27.963 INFO:teuthology.orchestra.run.smithi190.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:43:28.124 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:43:32.072 INFO:teuthology.orchestra.run.smithi190.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:43:32.419 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:43:32.552 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:43:33.993 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:44:01.874 INFO:teuthology.orchestra.run.smithi045.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:44:17.268 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:44:17.269 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-29T11:44:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:real 1m1.280s 2024-03-29T11:44:17.269 INFO:teuthology.orchestra.run.smithi045.stderr:user 2m44.451s 2024-03-29T11:44:17.270 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m5.284s 2024-03-29T11:44:25.151 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:44:25.153 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-29T11:44:25.153 INFO:teuthology.orchestra.run.smithi190.stderr:real 1m9.121s 2024-03-29T11:44:25.153 INFO:teuthology.orchestra.run.smithi190.stderr:user 1m56.311s 2024-03-29T11:44:25.153 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m4.361s 2024-03-29T11:44:25.153 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:44:25.154 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi045/log 2024-03-29T11:44:25.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:44:49.021 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi190/log 2024-03-29T11:44:49.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:45:06.892 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:07.126 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=06f3403e280d4923967cadc7a6eac2e3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:07.130 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:45:07.142 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:45:07.159 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:45:07.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:45:07.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:45:07.255 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:45:07.264 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:45:07.264 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:45:07.267 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:45:07.281 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:45:07.283 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +425us[ +429us] +/- 70ms 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -129us[ -125us] +/- 65ms 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -264us[ -264us] +/- 65ms 2024-03-29T11:45:07.286 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:45:07.288 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:45:07.288 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-29T11:45:07.288 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +421us[ +423us] +/- 70ms 2024-03-29T11:45:07.288 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -121us[ -119us] +/- 65ms 2024-03-29T11:45:07.288 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -253us[ -253us] +/- 65ms 2024-03-29T11:45:07.289 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:45:07.289 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:45:07.297 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:45:07.298 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:45:07.306 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:45:07.400 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:45:07.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:45:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:45:07.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:45:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:45:07.534 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:45:07.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:45:07.551 INFO:teuthology.task.internal:Duration was 1228.904367 seconds 2024-03-29T11:45:07.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:45:07.561 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:45:07.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:45:07.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:45:07.606 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:45:07.618 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:45:08.026 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:45:08.026 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-03-29T11:45:08.027 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:45:08.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-03-29T11:45:08.093 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:45:08.121 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:45:08.122 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:45:08.137 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:45:08.195 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:45:08.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:45:08.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:45:08.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:45:08.385 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:45:08.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:45:08.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:45:08.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:45:08.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:45:08.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:45:08.484 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-03-29T11:45:08.512 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-03-29T11:45:08.538 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:45:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:45:08.558 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:45:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:45:08.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:45:08.602 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:45:08.604 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi045 2024-03-29T11:45:08.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:45:08.640 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724/remote/smithi190 2024-03-29T11:45:08.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:45:08.675 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:45:08.676 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:45:08.679 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:45:08.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:45:08.736 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:45:08.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:45:08.771 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:45:08.772 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:45:08.774 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:45:08.786 INFO:teuthology.orchestra.run.smithi045.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:45 /home/ubuntu/cephtest 2024-03-29T11:45:08.788 INFO:teuthology.orchestra.run.smithi190.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:45 /home/ubuntu/cephtest 2024-03-29T11:45:08.789 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:45:08.801 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:45:08.826 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:45:08.852 DEBUG:teuthology.nuke:shortname: smithi045 2024-03-29T11:45:08.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:45:08.874 DEBUG:teuthology.nuke:shortname: smithi190 2024-03-29T11:45:08.874 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:45:08.877 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:42.275417', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTYl8mD0g8ZLBN36uwr+/8pyQ1eLH6j5qG+Sx0y7nd16YU3Vam4ldskPPXd7k656VJQ76Tk/gP4GQaHQDxxtpI='} 2024-03-29T11:45:08.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622724', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:42.276703', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCyO5V/BD/RsJnRH35tRBrBritxo2PqbBiQAiauzZgHMv0s0pwjuu5tEwE6RvCj+d/pKhHmHH9wYy3Nqxd1u0U0='} 2024-03-29T11:45:08.929 INFO:teuthology.orchestra.console.smithi045:Power off 2024-03-29T11:45:08.929 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:45:08.953 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-03-29T11:45:08.954 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:08.973 INFO:teuthology.orchestra.console.smithi190:Power off 2024-03-29T11:45:08.973 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:45:08.983 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-29T11:45:08.996 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-03-29T11:45:08.997 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:09.022 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-29T11:45:12.985 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:13.212 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:13.317 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-29T11:45:13.322 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-29T11:45:17.322 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:17.332 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:17.434 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-03-29T11:45:17.443 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-03-29T11:45:17.535 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-03-29T11:45:17.544 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-03-29T11:45:17.648 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1228.9043667316437 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=06f3403e280d4923967cadc7a6eac2e3 status: fail success: false 2024-03-29T11:45:17.649 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:45:17.725 INFO:teuthology.run:FAIL